terraform-provider-google icon indicating copy to clipboard operation
terraform-provider-google copied to clipboard

Published 20 hours ago • verified publisher icon hashicorp

Updating Internal regional Application Load Balancer while using NEG as a backend caused error.

Open SehiiRohoza opened this issue 1 year ago • 46 comments

Hello,

I have some issues while updating Internal regional Application Load Balancer while using NEG as a backend if endpoints were removed and added back NEG.

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

$ terraform  -v
Terraform v1.5.5

Affected Resource(s)

google_compute_region_backend_service

Terraform Configuration Files

data "google_compute_network_endpoint_group" "ilb_network_endpoint_group_zonal" {
  count   = var.environment == "qa" || var.environment == "test" || var.environment == "dev" ? 1 : 0
  name    = "name-${var.environment}-jira-neg"
  project = local.project_id
  zone    = "europe-west3-a"

  depends_on = [
    helm_release.jira
  ]
}

resource "google_compute_region_health_check" "ilb_health_check_zonal" {
  count   = var.environment == "qa" || var.environment == "test" || var.environment == "dev" ? 1 : 0
  name    = "name-${var.environment}-ilb-health-check"
  project = local.project_id
  region  = local.region
  
  timeout_sec         = 5
  check_interval_sec  = 5
  healthy_threshold   = 2
  unhealthy_threshold = 2

  http_health_check {
    port         = "8080"
    request_path = "/status"
    port_specification = "USE_FIXED_PORT"
  }
}

resource "google_compute_region_backend_service" "ilb_backend_service_zonal" {
  count                 = var.environment == "dev" || var.environment == "test" || var.environment == "qa" ? 1 : 0
  name                  = "name-${var.environment}-ilb-backend-service"
  project               = local.project_id
  region                = local.region
  health_checks         = [google_compute_region_health_check.ilb_health_check_zonal[0].id]
  protocol              = "HTTP"
  load_balancing_scheme = "INTERNAL_MANAGED"
  enable_cdn            = false
  session_affinity      = "GENERATED_COOKIE"
  locality_lb_policy    = "RING_HASH"
  timeout_sec           = 300

  backend {
    group                 = data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal[0].id
    balancing_mode        = "RATE"
    max_rate_per_endpoint = 1000
    capacity_scaler       = 1.0
  }

  consistent_hash {
    minimum_ring_size = 1024
  }
}

resource "google_compute_region_url_map" "ilb_url_map_zonal" {
  count           = var.environment == "dev" || var.environment == "test" || var.environment == "qa" ? 1 : 0
  name            = "name-${var.environment}-ilb-url-map"
  project         = local.project_id
  region          = local.region
  default_service = google_compute_region_backend_service.ilb_backend_service_zonal[0].id
}

resource "google_compute_region_target_http_proxy" "ilb_target_http_proxy_zonal" {
  count   = var.environment == "dev" || var.environment == "test" || var.environment == "qa" ? 1 : 0
  name    = "name-${var.environment}-ilb-https-proxy"
  project = local.project_id
  region  = local.region
  url_map = google_compute_region_url_map.ilb_url_map_zonal[0].id
}

resource "google_compute_forwarding_rule" "ilb_global_forwarding_rule_zonal" {
  count                 = var.environment == "dev" || var.environment == "test" || var.environment == "qa" ? 1 : 0
  name                  = "name-${var.environment}-ilb-global-forwarding-rule"
  project               = local.project_id
  region                = local.region
  network               = data.google_compute_network.network.self_link
  subnetwork            = data.google_compute_subnetwork.subnet.self_link
  ip_protocol           = "TCP"
  load_balancing_scheme = "INTERNAL_MANAGED"
  port_range            = "80"
  target                = google_compute_region_target_http_proxy.ilb_target_http_proxy_zonal[0].self_link  
  ip_address            = data.google_compute_address.nginx_ingress_ip.address
}

Expected Behavior

Internal regional Application Load Balancer should be updated.

Actual Behavior

There's an error message:


 Error: Provider produced inconsistent final plan
 
 When expanding the plan for
 google_compute_region_backend_service.ilb_backend_service_zonal[0] to
 include new values learned so far during apply, provider
 "registry.terraform.io/hashicorp/google" produced an invalid new value for
 .backend: planned set element
 cty.ObjectVal(map[string]cty.Value{"balancing_mode":cty.StringVal("RATE"),
 "capacity_scaler":cty.NumberIntVal(1), "description":cty.StringVal(""),
 "failover":cty.UnknownVal(cty.Bool), "group":cty.UnknownVal(cty.String),
 "max_connections":cty.NullVal(cty.Number),
 "max_connections_per_endpoint":cty.NullVal(cty.Number),
 "max_connections_per_instance":cty.NullVal(cty.Number),
 "max_rate":cty.NullVal(cty.Number),
 "max_rate_per_endpoint":cty.NumberIntVal(1000),
 "max_rate_per_instance":cty.NullVal(cty.Number),
 "max_utilization":cty.NullVal(cty.Number)}) does not correlate with any
 element in actual.
 
 This is a bug in the provider, which should be reported in the provider's
 own issue tracker.

There's no issue if run terraform apply again

Steps to Reproduce

  1. Create GKE service with NEG using annotations:
annotations:
      cloud.google.com/neg: '{"exposed_ports": {"80":{"name": "name-${environment}-jira-neg"}}}'
  1. Use Terraform code above to deploy Internal regional Application Load Balancer while using NEG as a backend.
  2. Update app parameters to trigger adding and removing endpoints to NEG and run terraform apply

Second execution of terraform apply shows no issues.

References

https://cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg https://cloud.google.com/load-balancing/docs/l7-internal

UPDATE

GCP Support confirmed that ILB is deployed fine and all commands that got triggered completed without errors for each build.

SehiiRohoza avatar Aug 28 '23 12:08 SehiiRohoza

@SehiiRohoza can you share your debug log that led to the error, and the 2nd apply that succeeded?

edwardmedia avatar Aug 28 '23 13:08 edwardmedia

@SehiiRohoza can you share your debug log that led to the error, and the 2nd apply that succeeded?

@edwardmedia sure, please find terraform apply below:

  • 1st run https://gist.github.com/SehiiRohoza/e6d5f4ebc5bf855061c7024ee0c1e04a
  • 2nd run https://gist.github.com/SehiiRohoza/da7a2a010dd7a22ae5a6153f90adb97f

I use quite the same approach for External HTTP(S) Load Balancer and it works without any issues.

SehiiRohoza avatar Aug 28 '23 14:08 SehiiRohoza

@SehiiRohoza I have reviewed both logs, and all api calls were GET requests. It was not failing on the provider when it tried to update. When you say Updating ..., what exactly was updated?

I noticed below statement. Did you see any other abnormal behavior, such as networking during that particular period?

State storage *remote.State declined to persist a state snapshot

Are you able to repro the issue? If yes, are you able to create a simplified config so I can run?

edwardmedia avatar Aug 28 '23 18:08 edwardmedia

@edwardmedia I haven't noticed any other abnormal behavior. Meanwhile, the Internal regional Application Load Balancer doesn't show the Health check which actually works (the backend NEG was marked as Heathy and I can't remove the Health check as it in use).

My configuration is quite complex as I'm deploying Jira Data Center using Helm chart https://github.com/atlassian/data-center-helm-charts/tree/main to GKE, CloudSQL and Filestore. To reproduce this issue, I made a simple change in values.yaml resulting in Helm chart redeployment (for example disable exposing of Jira JMX Monitoring). It could be tricky to reproduce on your side, but it's possible.

Alternatively, you can try to create some Deployment, expose it following https://cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg and create Internal regional Application Load Balancer while using NEG as a backend using my TF code above. Meanwhile, I'm not sure how to initiate an update to reproduce the issue.

SehiiRohoza avatar Aug 28 '23 19:08 SehiiRohoza

@SehiiRohoza the error I mentioned here leads me to think this is a temporally or external issue. Unless we can repro, it is hard to see what went to wrong.

edwardmedia avatar Aug 28 '23 19:08 edwardmedia

@edwardmedia that's strange as I had this error message a few times during a few days. Could it be related to some broken API calls on GCP side?

SehiiRohoza avatar Aug 28 '23 20:08 SehiiRohoza

@SehiiRohoza yes, it is possible. Sometimes, apis do experience difficulties. But I reviewed the logs, and all GET requested received 200. It is unlikely that was a broken call. To dig deeper to see if the issue is in the code, we need to make sure the issue is repeatable. You may want to investigate if there are other external reasons.

edwardmedia avatar Aug 28 '23 21:08 edwardmedia

@edwardmedia I have one open case with GCP support that could be related to it - there's no Health check shown while trying to edit Google Cloud Console the Internal regional Application Load Balancer created with a Terraform code above. Meanwhile, the command gcloud compute backend-services describe shows it in place and it actually works. There's a chance that fixing it on Google's side will fix this issue if it's related. In addition to that minimum_ring_size is optional for Terraform resource google_compute_region_backend_service meanwhile it's missing as well while trying to edit Google Cloud Console. I'll let you know the result of the case.

SehiiRohoza avatar Aug 29 '23 11:08 SehiiRohoza

Okay. Thanks. Waiting for the result

edwardmedia avatar Aug 29 '23 15:08 edwardmedia

@SehiiRohoza any updates?

edwardmedia avatar Sep 04 '23 16:09 edwardmedia

@SehiiRohoza any updates?

@edwardmedia I had an issue with a Google Cloud Console not showing Health check while creating by Terraform and it was fixed "UI considers a health check to be usable with NEGs if 'portSpecifiction' is 'USE_FIXED_PORT' or 'USE_SERVING_PORT'." Meanwhile, it's (Optional) at the Terraform documentation.

As for the issue with Terraform plan it is still in progress. I'll be back as soon as I get any update about it.

SehiiRohoza avatar Sep 04 '23 17:09 SehiiRohoza

@SehiiRohoza cool. Yes, let me know once you have the updates

edwardmedia avatar Sep 04 '23 19:09 edwardmedia

@edwardmedia GCP Support didn't find any issues on the GCP side. Th issue appears when a service detach and then attach back to NEG endpoints, for example:

jira    83s     Normal  Detach          service/jira    Detach 1 network endpoint(s) (NEG "name-test-jira-neg" in zone "europe-west3-a")
jira    69s     Normal  Attach          service/jira    Attach 1 network endpoint(s) (NEG "name-test-jira-neg" in zone "europe-west3-a")

after the data obtained Terraform there's an issue with google_compute_region_backend_service (error is in my first message). If you have 2 NEGs - you'll get 2 error messages. I tried to add a 360sec delay between updating NEG and reading data from it, but no luck. Using terraform-google-provider beta, also didn't help.

GCP Support suspects an issue at the Terraform side - terraform-google-provider. They confirmed that ILB is deployed fine and all commands that got triggered completed without errors for each build.

So, we're back from what we started. I updated google_compute_region_health_check in first message and reproduction steps.

SehiiRohoza avatar Sep 07 '23 17:09 SehiiRohoza

I tried to run Terraform from my laptop - the issue is the same:

helm_release.jira: Modifications complete after 4m25s [id=jira]
data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal[0]: Reading...
data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal[0]: Read complete after 0s 
╷
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for google_compute_region_backend_service.ilb_backend_service_zonal[0] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google" produced
│ an invalid new value for .backend: planned set element cty.ObjectVal(map[string]cty.Value{"balancing_mode":cty.StringVal("RATE"), "capacity_scaler":cty.NumberIntVal(1), "description":cty.StringVal(""),
│ "failover":cty.UnknownVal(cty.Bool), "group":cty.UnknownVal(cty.String), "max_connections":cty.NullVal(cty.Number), "max_connections_per_endpoint":cty.NullVal(cty.Number),
│ "max_connections_per_instance":cty.NullVal(cty.Number), "max_rate":cty.NullVal(cty.Number), "max_rate_per_endpoint":cty.NumberIntVal(2000), "max_rate_per_instance":cty.NullVal(cty.Number),
│ "max_utilization":cty.NullVal(cty.Number)}) does not correlate with any element in actual.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

SehiiRohoza avatar Sep 08 '23 13:09 SehiiRohoza

@shuyama1 any idea what is happening?

edwardmedia avatar Sep 11 '23 15:09 edwardmedia

@edwardmedia @shuyama1 I think I was able to find a work around.

I've changed the code as below:

resource "google_compute_region_backend_service" "ilb_backend_service_zonal" {
  count                 = var.environment == "dev" || var.environment == "test" || var.environment == "qa" ? 1 : 0
  name                  = "name-${var.environment}-ilb-backend-service"
  project               = local.project_id
  region                = local.region
  health_checks         = [google_compute_region_health_check.ilb_health_check_zonal[0].id]
  protocol              = "HTTP"
  load_balancing_scheme = "INTERNAL_MANAGED"
  enable_cdn            = false
  session_affinity      = "GENERATED_COOKIE"
  locality_lb_policy    = "RING_HASH"
  timeout_sec           = 300

  backend {
    group           = "projects/${local.project_id}/zones/europe-west3-a/networkEndpointGroups/name-${var.environment}-jira-neg"
    #group                 = data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal[0].id
    balancing_mode        = "RATE"
    max_rate_per_endpoint = 1000
    capacity_scaler       = 1.0
  }

  consistent_hash {
    minimum_ring_size = 1024
  }
  
  depends_on = [
    helm_release.jira
  ]
}

and it works like a charm:

Step #3 - "tf apply": helm_release.jira: Still modifying... [id=jira, 5m10s elapsed]
Step #3 - "tf apply": helm_release.jira: Modifications complete after 5m14s [id=jira]
Step #3 - "tf apply": data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal[0]: Reading...
Step #3 - "tf apply": data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal[0]: Read complete after 0s [id=projects/project-id-test/zones/europe-west3-a/networkEndpointGroups/name-test-jira-neg]
Step #3 - "tf apply": 
Step #3 - "tf apply": Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

After that, I checked tfstate and found that there's no difference:

    {
      "mode": "data",
      "type": "google_compute_network_endpoint_group",
      "name": "ilb_network_endpoint_group_zonal",
      "provider": "provider[\"registry.terraform.io/hashicorp/google\"]",
      "instances": [
        {
          "index_key": 0,
          "schema_version": 0,
          "attributes": {
            "default_port": 0,
            "description": "{\"cluster-uid\":\"5b267696-062b-44ae-b771-a1946ce02abf\",\"namespace\":\"jira\",\"service-name\":\"jira\",\"port\":\"80\"}",
            "id": "projects/project-id-test/zones/europe-west3-a/networkEndpointGroups/name-test-jira-neg",
            "name": "name-test-jira-neg",
            "network": "https://www.googleapis.com/compute/v1/projects/project-id-test/global/networks/name-test-vpc",
            "network_endpoint_type": "GCE_VM_IP_PORT",
            "project": "project-id-test",
            "self_link": "https://www.googleapis.com/compute/v1/projects/project-id-test/zones/europe-west3-a/networkEndpointGroups/name-test-jira-neg",
            "size": 1,
            "subnetwork": "https://www.googleapis.com/compute/v1/projects/project-id-test/regions/europe-west3/subnetworks/name-test-subnet",
            "zone": "https://www.googleapis.com/compute/v1/projects/project-id-test/zones/europe-west3-a"
          },
          "sensitive_attributes": []
        }
      ]
    },

Meanwhile, when I change it back as below:

resource "google_compute_region_backend_service" "ilb_backend_service_zonal" {
  count                 = var.environment == "dev" || var.environment == "test" || var.environment == "qa" ? 1 : 0
  name                  = "name-${var.environment}-ilb-backend-service"
  project               = local.project_id
  region                = local.region
  health_checks         = [google_compute_region_health_check.ilb_health_check_zonal[0].id]
  protocol              = "HTTP"
  load_balancing_scheme = "INTERNAL_MANAGED"
  enable_cdn            = false
  session_affinity      = "GENERATED_COOKIE"
  locality_lb_policy    = "RING_HASH"
  timeout_sec           = 300

  backend {
    #group           = "projects/${local.project_id}/zones/europe-west3-a/networkEndpointGroups/name-${var.environment}-jira-neg"
    group                 = data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal[0].id
    balancing_mode        = "RATE"
    max_rate_per_endpoint = 1000
    capacity_scaler       = 1.0
  }

  consistent_hash {
    minimum_ring_size = 1024
  }
  
  depends_on = [
    helm_release.jira
  ]
}

the issue was back as well:

Step #3 - "tf apply": helm_release.jira: Modifications complete after 5m38s [id=jira]
Step #3 - "tf apply": data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal[0]: Reading...
Step #3 - "tf apply": data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal[0]: Read complete after 1s [id=projects/project-id-test/zones/europe-west3-a/networkEndpointGroups/name-test-jira-neg]
Step #3 - "tf apply": ╷
Step #3 - "tf apply": │ Error: Provider produced inconsistent final plan
Step #3 - "tf apply": │ 
Step #3 - "tf apply": │ When expanding the plan for
Step #3 - "tf apply": │ google_compute_region_backend_service.ilb_backend_service_zonal[0] to
Step #3 - "tf apply": │ include new values learned so far during apply, provider
Step #3 - "tf apply": │ "registry.terraform.io/hashicorp/google" produced an invalid new value for
Step #3 - "tf apply": │ .backend: planned set element
Step #3 - "tf apply": │ cty.ObjectVal(map[string]cty.Value{"balancing_mode":cty.StringVal("RATE"),
Step #3 - "tf apply": │ "capacity_scaler":cty.NumberIntVal(1), "description":cty.StringVal(""),
Step #3 - "tf apply": │ "failover":cty.UnknownVal(cty.Bool), "group":cty.UnknownVal(cty.String),
Step #3 - "tf apply": │ "max_connections":cty.NullVal(cty.Number),
Step #3 - "tf apply": │ "max_connections_per_endpoint":cty.NullVal(cty.Number),
Step #3 - "tf apply": │ "max_connections_per_instance":cty.NullVal(cty.Number),
Step #3 - "tf apply": │ "max_rate":cty.NullVal(cty.Number),
Step #3 - "tf apply": │ "max_rate_per_endpoint":cty.NumberIntVal(2000),
Step #3 - "tf apply": │ "max_rate_per_instance":cty.NullVal(cty.Number),
Step #3 - "tf apply": │ "max_utilization":cty.NullVal(cty.Number)}) does not correlate with any
Step #3 - "tf apply": │ element in actual.
Step #3 - "tf apply": │ 
Step #3 - "tf apply": │ This is a bug in the provider, which should be reported in the provider's
Step #3 - "tf apply": │ own issue tracker.
Step #3 - "tf apply": ╵
Finished Step #3 - "tf apply"

SehiiRohoza avatar Sep 11 '23 18:09 SehiiRohoza

In addition to that, I conducted one more experiment. I changed the configuration as below:

resource "google_compute_region_backend_service" "ilb_backend_service_zonal" {
  count                 = var.environment == "dev" || var.environment == "test" || var.environment == "qa" ? 1 : 0
  name                  = "name-${var.environment}-ilb-backend-service"
  project               = local.project_id
  region                = local.region
  health_checks         = [google_compute_region_health_check.ilb_health_check_zonal[0].id]
  protocol              = "HTTP"
  load_balancing_scheme = "INTERNAL_MANAGED"
  enable_cdn            = false
  session_affinity      = "GENERATED_COOKIE"
  locality_lb_policy    = "RING_HASH"
  timeout_sec           = 300

  backend {
    group           = "https://www.googleapis.com/compute/v1/projects/${local.project_id}/zones/europe-west3-a/networkEndpointGroups/name-${var.environment}-jira-neg"
    #group                 = data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal[0].id
    balancing_mode        = "RATE"
    max_rate_per_endpoint = 1000
    capacity_scaler       = 1.0
  }

  consistent_hash {
    minimum_ring_size = 1024
  }
  
  depends_on = [
    helm_release.jira
  ]
}

and it works like a charm as well.

Meanwhile, when I tried to get back to data as below (which is quite the same):

resource "google_compute_region_backend_service" "ilb_backend_service_zonal" {
  count                 = var.environment == "dev" || var.environment == "test" || var.environment == "qa" ? 1 : 0
  name                  = "name-${var.environment}-ilb-backend-service"
  project               = local.project_id
  region                = local.region
  health_checks         = [google_compute_region_health_check.ilb_health_check_zonal[0].id]
  protocol              = "HTTP"
  load_balancing_scheme = "INTERNAL_MANAGED"
  enable_cdn            = false
  session_affinity      = "GENERATED_COOKIE"
  locality_lb_policy    = "RING_HASH"
  timeout_sec           = 300

  backend {
    #group           = "https://www.googleapis.com/compute/v1/projects/${local.project_id}/zones/europe-west3-a/networkEndpointGroups/name-${var.environment}-jira-neg"
    group                 = data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal[0].self_link
    balancing_mode        = "RATE"
    max_rate_per_endpoint = 1000
    capacity_scaler       = 1.0
  }

  consistent_hash {
    minimum_ring_size = 1024
  }
  
  depends_on = [
    helm_release.jira
  ]
}

I got an error message:

Step #2 - "tf plan": ╷
Step #2 - "tf plan": │ Error: Missing required argument
Step #2 - "tf plan": │ 
Step #2 - "tf plan": │   with google_compute_region_backend_service.ilb_backend_service_zonal[0],
Step #2 - "tf plan": │   on lb.tf line 47, in resource "google_compute_region_backend_service" "ilb_backend_service_zonal":
Step #2 - "tf plan": │   47: resource "google_compute_region_backend_service" "ilb_backend_service_zonal" {
Step #2 - "tf plan": │ 
Step #2 - "tf plan": │ The argument "backend.0.group" is required, but no definition was found.
Step #2 - "tf plan": ╵

I hope that this will be helpful in your investigation.

SehiiRohoza avatar Sep 11 '23 18:09 SehiiRohoza

Hey Terraform Team, any updates on this?

mikolaj-stoch avatar Sep 22 '23 11:09 mikolaj-stoch

It looks like we never got a minimal reproducible case on this ticket - that is, a configuration that contains all information necessary to cause the behavior (not using data sources or variables.) If anyone is able to provide one that would be super helpful!

melinath avatar Nov 16 '23 23:11 melinath

I am also experiencing this behaviour our of a sudden.

Ill attach the logs below.

I am running terraform on a remote dev instance hosted in GCP I also first encountered this issue when running on Gitlab CI pipeline

is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .api_config: was cty.StringVal("projects/{{redacted}}/locations/global/apis/dev-ls-receipt-api/configs/ls-api-z74"), but now cty.StringVal("projects/{{redacted}}/locations/global/apis/dev-ls-receipt-api/configs/ls-api-z74")
2024-01-22T05:53:51.884Z [DEBUG] states/remote: state read serial is: 125; serial is: 125
2024-01-22T05:53:51.884Z [DEBUG] states/remote: state read lineage is: e95e323a-734d-ae1f-77d0-f0c8ac4a4e2e; lineage is: e95e323a-734d-ae1f-77d0-f0c8ac4a4e2e
2024-01-22T05:53:52.069Z [INFO]  Starting apply for module.api_gateway["us-central1"].google_compute_region_network_endpoint_group.apigateway_neg
2024-01-22T05:53:52.070Z [DEBUG] module.api_gateway["us-central1"].google_compute_region_network_endpoint_group.apigateway_neg: applying the planned Create change
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Creating new RegionNetworkEndpointGroup: map[string]interface {}{"name":"dev-us-central1-api-gateway-neg", "networkEndpointType":"SERVERLESS", "region":"projects/{{redacted}}/global/regions/us-central1", "serverlessDeployment":map[string]interface {}{"platform":"apigateway.googleapis.com", "resource":"dev-us-central1-ls-receipts-gateway"}}
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Waiting for state to become: [success]
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Retry Transport: starting RoundTrip retry loop
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Retry Transport: request attempt 0
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Google API Request Details:
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: ---[ REQUEST ]---------------------------------------
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: POST /compute/beta/projects/{{redacted}}/regions/us-central1/networkEndpointGroups?alt=json HTTP/1.1
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Host: compute.googleapis.com
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: User-Agent: Terraform/1.7.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google-beta/5.10.0
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Content-Length: 251
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Content-Type: application/json
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Accept-Encoding: gzip
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: {
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "name": "dev-us-central1-api-gateway-neg",
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "networkEndpointType": "SERVERLESS",
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "region": "projects/{{redacted}}/global/regions/us-central1",
2024-01-22T05:53:52.073Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "serverlessDeployment": {
2024-01-22T05:53:52.074Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "platform": "apigateway.googleapis.com",
2024-01-22T05:53:52.074Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "resource": "dev-us-central1-ls-receipts-gateway"
2024-01-22T05:53:52.074Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  }
2024-01-22T05:53:52.074Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: }
2024-01-22T05:53:52.074Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:53:52.074Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: -----------------------------------------------------
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Google API Response Details:
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: ---[ RESPONSE ]--------------------------------------
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: HTTP/2.0 200 OK
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Cache-Control: private
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Content-Type: application/json; charset=UTF-8
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Date: Mon, 22 Jan 2024 05:53:52 GMT
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Server: ESF
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Vary: Origin
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Vary: X-Origin
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Vary: Referer
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Content-Type-Options: nosniff
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Frame-Options: SAMEORIGIN
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Xss-Protection: 0
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: {
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "kind": "compute#operation",
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "id": "5069853238413736991",
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "name": "operation-1705902832088-60f827453f437-4aafc554-1e5a2b10",
2024-01-22T05:53:52.355Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "operationType": "compute.regionNetworkEndpointGroups.insert",
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "targetLink": "https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1/networkEndpointGroups/dev-us-central1-api-gateway-neg",
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "targetId": "6991214352997268511",
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "status": "RUNNING",
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "user": "{{user_email_redacted}}",
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "progress": 0,
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "insertTime": "2024-01-21T21:53:52.252-08:00",
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "startTime": "2024-01-21T21:53:52.264-08:00",
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "selfLink": "https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1/operations/operation-1705902832088-60f827453f437-4aafc554-1e5a2b10",
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "region": "https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1"
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: }
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: -----------------------------------------------------
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Retry Transport: Stopping retries, last request was successful
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Retry Transport: Returning after 1 attempts
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [INFO] Instantiating GCE client for path https://compute.googleapis.com/compute/beta/
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Waiting for state to become: [DONE]
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Retry Transport: starting RoundTrip retry loop
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Retry Transport: request attempt 0
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Google API Request Details:
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: ---[ REQUEST ]---------------------------------------
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: GET /compute/beta/projects/{{redacted}}/regions/us-central1/operations/operation-1705902832088-60f827453f437-4aafc554-1e5a2b10?alt=json&prettyPrint=false HTTP/1.1
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Host: compute.googleapis.com
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.7.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google-beta/5.10.0
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Goog-Api-Client: gl-go/1.20.12 gdcl/0.152.0
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Accept-Encoding: gzip
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:53:52.356Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: -----------------------------------------------------
2024-01-22T05:53:52.433Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Google API Response Details:
2024-01-22T05:53:52.434Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: ---[ RESPONSE ]--------------------------------------
2024-01-22T05:53:52.434Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: HTTP/2.0 200 OK
2024-01-22T05:53:52.434Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Cache-Control: private
2024-01-22T05:53:52.434Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Content-Type: application/json; charset=UTF-8
2024-01-22T05:53:52.434Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Date: Mon, 22 Jan 2024 05:53:52 GMT
2024-01-22T05:53:52.434Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Etag: 8IVhBI8eeHlRKJdj3IaDIiWgpqc=/3wLtadrq3CtJDBldMAFgMON9KBA=
2024-01-22T05:53:52.434Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Server: ESF
2024-01-22T05:53:52.435Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Vary: Origin
2024-01-22T05:53:52.435Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Vary: X-Origin
2024-01-22T05:53:52.435Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Vary: Referer
2024-01-22T05:53:52.435Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Content-Type-Options: nosniff
2024-01-22T05:53:52.435Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Frame-Options: SAMEORIGIN
2024-01-22T05:53:52.435Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Xss-Protection: 0
2024-01-22T05:53:52.435Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:53:52.435Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: {
2024-01-22T05:53:52.436Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "kind": "compute#operation",
2024-01-22T05:53:52.436Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "id": "5069853238413736991",
2024-01-22T05:53:52.436Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "name": "operation-1705902832088-60f827453f437-4aafc554-1e5a2b10",
2024-01-22T05:53:52.436Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "operationType": "compute.regionNetworkEndpointGroups.insert",
2024-01-22T05:53:52.436Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "targetLink": "https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1/networkEndpointGroups/dev-us-central1-api-gateway-neg",
2024-01-22T05:53:52.437Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "targetId": "6991214352997268511",
2024-01-22T05:53:52.437Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "status": "RUNNING",
2024-01-22T05:53:52.437Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "user": "{{user_email_redacted}}",
2024-01-22T05:53:52.437Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "progress": 0,
2024-01-22T05:53:52.437Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "insertTime": "2024-01-21T21:53:52.252-08:00",
2024-01-22T05:53:52.437Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "startTime": "2024-01-21T21:53:52.264-08:00",
2024-01-22T05:53:52.437Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "selfLink": "https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1/operations/operation-1705902832088-60f827453f437-4aafc554-1e5a2b10",
2024-01-22T05:53:52.437Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "region": "https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1"
2024-01-22T05:53:52.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: }
2024-01-22T05:53:52.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: -----------------------------------------------------
2024-01-22T05:53:52.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Retry Transport: Stopping retries, last request was successful
2024-01-22T05:53:52.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Retry Transport: Returning after 1 attempts
2024-01-22T05:53:52.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [DEBUG] Got RUNNING while polling for operation operation-1705902832088-60f827453f437-4aafc554-1e5a2b10's status
2024-01-22T05:53:52.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:53:52 [TRACE] Waiting 10s before next try
2024-01-22T05:54:02.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Retry Transport: starting RoundTrip retry loop
2024-01-22T05:54:02.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Retry Transport: request attempt 0
2024-01-22T05:54:02.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Google API Request Details:
2024-01-22T05:54:02.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: ---[ REQUEST ]---------------------------------------
2024-01-22T05:54:02.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: GET /compute/beta/projects/{{redacted}}/regions/us-central1/operations/operation-1705902832088-60f827453f437-4aafc554-1e5a2b10?alt=json&prettyPrint=false HTTP/1.1
2024-01-22T05:54:02.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Host: compute.googleapis.com
2024-01-22T05:54:02.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.7.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google-beta/5.10.0
2024-01-22T05:54:02.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Goog-Api-Client: gl-go/1.20.12 gdcl/0.152.0
2024-01-22T05:54:02.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Accept-Encoding: gzip
2024-01-22T05:54:02.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:54:02.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:54:02.438Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: -----------------------------------------------------
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Google API Response Details:
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: ---[ RESPONSE ]--------------------------------------
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: HTTP/2.0 200 OK
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Cache-Control: private
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Content-Type: application/json; charset=UTF-8
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Date: Mon, 22 Jan 2024 05:54:02 GMT
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Etag: jGcjX55syFCRcOE1zwDBo8hUJpA=/r0U_TFXgY6M6nDQRZZUYSd-HF54=
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Server: ESF
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Vary: Origin
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Vary: X-Origin
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Vary: Referer
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Content-Type-Options: nosniff
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Frame-Options: SAMEORIGIN
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Xss-Protection: 0
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: {
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "kind": "compute#operation",
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "id": "5069853238413736991",
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "name": "operation-1705902832088-60f827453f437-4aafc554-1e5a2b10",
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "operationType": "compute.regionNetworkEndpointGroups.insert",
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "targetLink": "https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1/networkEndpointGroups/dev-us-central1-api-gateway-neg",
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "targetId": "6991214352997268511",
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "status": "DONE",
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "user": "{{user_email_redacted}}",
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "progress": 100,
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "insertTime": "2024-01-21T21:53:52.252-08:00",
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "startTime": "2024-01-21T21:53:52.264-08:00",
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "endTime": "2024-01-21T21:53:52.713-08:00",
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "selfLink": "https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1/operations/operation-1705902832088-60f827453f437-4aafc554-1e5a2b10",
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:  "region": "https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1"
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: }
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: -----------------------------------------------------
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Retry Transport: Stopping retries, last request was successful
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Retry Transport: Returning after 1 attempts
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Got DONE while polling for operation operation-1705902832088-60f827453f437-4aafc554-1e5a2b10's status
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Finished creating RegionNetworkEndpointGroup "projects/{{redacted}}/regions/us-central1/networkEndpointGroups/dev-us-central1-api-gateway-neg": map[string]interface {}{"id":"5069853238413736991", "insertTime":"2024-01-21T21:53:52.252-08:00", "kind":"compute#operation", "name":"operation-1705902832088-60f827453f437-4aafc554-1e5a2b10", "operationType":"compute.regionNetworkEndpointGroups.insert", "progress":0, "region":"https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1", "selfLink":"https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1/operations/operation-1705902832088-60f827453f437-4aafc554-1e5a2b10", "startTime":"2024-01-21T21:53:52.264-08:00", "status":"RUNNING", "targetId":"6991214352997268511", "targetLink":"https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1/networkEndpointGroups/dev-us-central1-api-gateway-neg", "user":"{{user_email_redacted}}"}
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Waiting for state to become: [success]
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Retry Transport: starting RoundTrip retry loop
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Retry Transport: request attempt 0
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Google API Request Details:
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: ---[ REQUEST ]---------------------------------------
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: GET /compute/beta/projects/{{redacted}}/regions/us-central1/networkEndpointGroups/dev-us-central1-api-gateway-neg?alt=json HTTP/1.1
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Host: compute.googleapis.com
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: User-Agent: Terraform/1.7.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google-beta/5.10.0
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Content-Type: application/json
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Accept-Encoding: gzip
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:54:02.530Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: -----------------------------------------------------
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Google API Response Details:
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: ---[ RESPONSE ]--------------------------------------
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: HTTP/2.0 200 OK
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Cache-Control: private
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Content-Type: application/json; charset=UTF-8
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Date: Mon, 22 Jan 2024 05:54:02 GMT
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Etag: mWP6GwbrkLOOenp48DBs_eBy0J4=/bsNJBi_qbVmArLnIVTQdhI01VK0=
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Server: ESF
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Vary: Origin
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Vary: X-Origin
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: Vary: Referer
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Content-Type-Options: nosniff
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Frame-Options: SAMEORIGIN
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: X-Xss-Protection: 0
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: {
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "kind": "compute#networkEndpointGroup",
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "id": "6991214352997268511",
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "creationTimestamp": "2024-01-21T21:53:52.258-08:00",
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "selfLink": "https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1/networkEndpointGroups/dev-us-central1-api-gateway-neg",
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "name": "dev-us-central1-api-gateway-neg",
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "networkEndpointType": "SERVERLESS",
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "size": 0,
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "region": "https://www.googleapis.com/compute/beta/projects/{{redacted}}/regions/us-central1",
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   "serverlessDeployment": {
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:     "platform": "apigateway.googleapis.com",
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:     "resource": "dev-us-central1-ls-receipts-gateway"
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5:   }
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: }
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: -----------------------------------------------------
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Retry Transport: Stopping retries, last request was successful
2024-01-22T05:54:02.620Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] Retry Transport: Returning after 1 attempts
2024-01-22T05:54:02.621Z [WARN]  Provider "provider[\"registry.terraform.io/hashicorp/google-beta\"]" produced an unexpected new value for module.api_gateway["us-central1"].google_compute_region_network_endpoint_group.apigateway_neg, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .region: was cty.StringVal("us-central1"), but now cty.StringVal("https://www.googleapis.com/compute/v1/projects/{{redacted}}/regions/us-central1")
      - .description: was null, but now cty.StringVal("")
      - .network: was null, but now cty.StringVal("")
      - .psc_target_service: was null, but now cty.StringVal("")
      - .subnetwork: was null, but now cty.StringVal("")
      - .serverless_deployment[0].url_mask: was null, but now cty.StringVal("")
      - .serverless_deployment[0].version: was null, but now cty.StringVal("")
2024-01-22T05:54:02.622Z [DEBUG] State storage *remote.State declined to persist a state snapshot
2024-01-22T05:54:02.626Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] [transport] [server-transport 0xc0012fc000] Closing: Server.Stop called
2024-01-22T05:54:02.626Z [DEBUG] provider.terraform-provider-google-beta_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] [transport] [server-transport 0xc0012fc000] loopyWriter exiting with error: transport closed by client
2024-01-22T05:54:02.627Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-01-22T05:54:02.628Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google-beta/5.10.0/linux_amd64/terraform-provider-google-beta_v5.10.0_x5 pid=2065273
2024-01-22T05:54:02.629Z [DEBUG] provider: plugin exited
2024-01-22T05:54:02.635Z [DEBUG] provider.terraform-provider-google_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] hashing map[balancing_mode:UTILIZATION capacity_scaler:1 description:backend NEG for https://www.googleapis.com/compute/v1/projects/{{redacted}}/regions/us-central1 group:projects/{{redacted}}/regions/us-central1/networkEndpointGroups/dev-us-central1-api-gateway-neg max_connections:0 max_connections_per_endpoint:0 max_connections_per_instance:0 max_rate:0 max_rate_per_endpoint:0 max_rate_per_instance:0 max_utilization:0]
2024-01-22T05:54:02.635Z [DEBUG] provider.terraform-provider-google_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] writing description backend NEG for https://www.googleapis.com/compute/v1/projects/{{redacted}}/regions/us-central1
2024-01-22T05:54:02.635Z [DEBUG] provider.terraform-provider-google_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] computed hash value of 676366148 from projects/{{redacted}}/regions/us-central1/networkEndpointGroups/dev-us-central1-api-gateway-neg-UTILIZATION-1.000000-backend NEG for https://www.googleapis.com/compute/v1/projects/{{redacted}}/regions/us-central1-0-0.000000-0-0-0.000000-0-0.000000-
2024-01-22T05:54:02.643Z [WARN]  Provider "registry.terraform.io/hashicorp/google" produced an invalid plan for module.load_balancer_default.google_compute_backend_service.ls_loadbalancer_default_backend, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .connection_draining_timeout_sec: planned value cty.NumberIntVal(300) for a non-computed attribute
      - .log_config: attribute representing nested block must not be unknown itself; set nested attribute values to unknown instead
      - .cdn_policy: attribute representing nested block must not be unknown itself; set nested attribute values to unknown instead
2024-01-22T05:54:02.644Z [ERROR] vertex "module.load_balancer_default.google_compute_backend_service.ls_loadbalancer_default_backend" error: Provider produced inconsistent final plan
2024-01-22T05:54:02.645Z [DEBUG] states/remote: state read serial is: 126; serial is: 126
2024-01-22T05:54:02.645Z [DEBUG] states/remote: state read lineage is: e95e323a-734d-ae1f-77d0-f0c8ac4a4e2e; lineage is: e95e323a-734d-ae1f-77d0-f0c8ac4a4e2e
╷
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for module.load_balancer_default.google_compute_backend_service.ls_loadbalancer_default_backend to include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/google" produced an invalid new value for .backend: planned set element
│ cty.ObjectVal(map[string]cty.Value{"balancing_mode":cty.StringVal("UTILIZATION"), "capacity_scaler":cty.NumberIntVal(1), "description":cty.StringVal("backend NEG for us-central1"),
│ "group":cty.UnknownVal(cty.String), "max_connections":cty.UnknownVal(cty.Number), "max_connections_per_endpoint":cty.UnknownVal(cty.Number),
│ "max_connections_per_instance":cty.UnknownVal(cty.Number), "max_rate":cty.UnknownVal(cty.Number), "max_rate_per_endpoint":cty.UnknownVal(cty.Number),
│ "max_rate_per_instance":cty.UnknownVal(cty.Number), "max_utilization":cty.UnknownVal(cty.Number)}) does not correlate with any element in actual.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
2024-01-22T05:54:02.984Z [DEBUG] provider.terraform-provider-google_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] [transport] [server-transport 0xc000cd2000] Closing: Server.Stop called
2024-01-22T05:54:02.984Z [DEBUG] provider.terraform-provider-google_v5.10.0_x5: 2024/01/22 05:54:02 [DEBUG] [transport] [server-transport 0xc000cd2000] loopyWriter exiting with error: transport closed by client
2024-01-22T05:54:02.985Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-01-22T05:54:02.987Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.10.0/linux_amd64/terraform-provider-google_v5.10.0_x5 pid=2065280
2024-01-22T05:54:02.988Z [DEBUG] provider: plugin exited



resource "google_api_gateway_api" "default" {
  depends_on = [
    module.enable_apis
  ]
  provider     = google-beta
  api_id       = "dev-ls-receipt-api"
  display_name = "dev-ls-receipt-api"
}


resource "google_api_gateway_api_config" "ls_api_config" {
  provider = google-beta
  api      = google_api_gateway_api.default.api_gateway_id
  api_config_id = "ls-api-${random_string.name.result}"

  openapi_documents {
    document {
      path     = local.api_schema
      contents = base64encode("${IMPORT ANY OPENSPEC1.0 HERE}")
      # contents = ""
    }
  }
  gateway_config {
    backend_config {
      google_service_account = "{{service account}}"
    }
  }
  lifecycle {
    create_before_destroy = true
  }
}

resource "google_api_gateway_gateway" "ls_api_gateway" {
  provider   = google-beta
  api_config = google_api_gateway_api_config.ls_api_config.id
  gateway_id = "dev-us-central-1-ls-receipts-gateway"
  region     = "us-central-1"
}

resource "google_compute_region_network_endpoint_group" "apigateway_neg" {
  provider              = google-beta
  name                  = "dev-api-gateway-neg"
  network_endpoint_type = "SERVERLESS"
  region                = var.service_region
  serverless_deployment {
    platform = "apigateway.googleapis.com"
    resource = google_api_gateway_gateway.ls_api_gateway.gateway_id
  }
  lifecycle {
    create_before_destroy = true
  }
}



resource "google_compute_backend_service" "ls_loadbalancer_default_backend" {
  name        = "dev-backend-services"
  enable_cdn  = false

  locality_lb_policy    = "ROUND_ROBIN"
  load_balancing_scheme = "EXTERNAL_MANAGED"
  protocol              = "HTTPS"

  dynamic "backend" {
    for_each = var.serverless_negs
    content {
      description = "backend NEG for ${backend.value.neg.region}"
      group = backend.value.neg.id
      # balancing_mode  = "UTILIZATION"
      # capacity_scaler = 1
      # max_connections = null
      # max_connections_per_endpoint = null
      # max_connections_per_instance = null
      # max_rate = null
      # max_rate_per_endpoint = 
      # max_rate_per_instance = null
      # max_utilization = 1
    }
  }

  lifecycle {
    create_before_destroy = true
  }
}

Please do note, that I am creating multiple regional API gateway, and creating a dynamic block backend for my LB

As others have mentioned on this thread, applying terraform again does create all the remaining resources (ALB too), and get success.

terraform -v
Terraform v1.7.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v5.10.0
+ provider registry.terraform.io/hashicorp/google-beta v5.10.0
+ provider registry.terraform.io/hashicorp/random v3.6.0
+ provider registry.terraform.io/hashicorp/time v0.10.0

nakiami10 avatar Jan 22 '24 06:01 nakiami10

For some reason, it was workin on google provider version 5.4.0 before, but somewhat changed after updating to terraform 1.7.0.

I feel it has something to do with the dynamic block?

nakiami10 avatar Jan 22 '24 18:01 nakiami10

Hello,

FYI, as update. I have skipped using dynamic block and instead hardcoded, it worked without issue. seems the bug is dynamic block compatibility with google provider?

  backend {
    description = "backend NEG for us-central1"
    group       = var.serverless_negs["us-central1"].network_endpoints_group.id
  }

nakiami10 avatar Jan 22 '24 19:01 nakiami10

I've come up with a workaround.

I've added a locals on the module where the load balancer is calling with the dynamic backend.

locals {
  negs = { for region, attr in var.serverless_negs : region => attr.network_endpoints_group.id }
}

nakiami10 avatar Jan 22 '24 21:01 nakiami10

@SehiiRohoza is this still an issue with your case? Keep in mind adding dynamic code is beyond what this repo handles. Please share a simple config if your issue can be repro'ed

edwardmedia avatar Jan 26 '24 17:01 edwardmedia

@SehiiRohoza is this still an issue with your case? Keep in mind adding dynamic code is beyond what this repo handles. Please share a simple config if your issue can be repro'ed

@edwardmedia I provided my exact configuration in the very first post, later I added debug and other information. Please let me know what else should I provide as everything above is still valid. It is still reproducible if you give it a try.

SehiiRohoza avatar Jan 26 '24 17:01 SehiiRohoza

@SehiiRohoza reviewed your configs here, the difference appears to be data from between hard-coded and the data source.

    #group           = "https://www.googleapis.com/compute/v1/projects/${local.project_id}/zones/europe-west3-a/networkEndpointGroups/name-${var.environment}-jira-neg"
    group                 = data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal[0].self_link

The hard-coded works but the data source doesn't. Based on below error, have you checked if the data source returns a valid value?

Step #2 - "tf plan": ╷
Step #2 - "tf plan": │ Error: Missing required argument
Step #2 - "tf plan": │ 
Step #2 - "tf plan": │   with google_compute_region_backend_service.ilb_backend_service_zonal[0],
Step #2 - "tf plan": │   on lb.tf line 47, in resource "google_compute_region_backend_service" "ilb_backend_service_zonal":
Step #2 - "tf plan": │   47: resource "google_compute_region_backend_service" "ilb_backend_service_zonal" {
Step #2 - "tf plan": │ 
Step #2 - "tf plan": │ The argument "backend.0.group" is required, but no definition was found.
Step #2 - "tf plan": ╵

edwardmedia avatar Jan 28 '24 16:01 edwardmedia

@SehiiRohoza reviewed your configs here, the difference appears to be data from between hard-coded and the data source.

    #group           = "https://www.googleapis.com/compute/v1/projects/${local.project_id}/zones/europe-west3-a/networkEndpointGroups/name-${var.environment}-jira-neg"
    group                 = data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal[0].self_link

The hard-coded works but the data source doesn't. Based on below error, have you checked if the data source returns a valid value?

Step #2 - "tf plan": ╷
Step #2 - "tf plan": │ Error: Missing required argument
Step #2 - "tf plan": │ 
Step #2 - "tf plan": │   with google_compute_region_backend_service.ilb_backend_service_zonal[0],
Step #2 - "tf plan": │   on lb.tf line 47, in resource "google_compute_region_backend_service" "ilb_backend_service_zonal":
Step #2 - "tf plan": │   47: resource "google_compute_region_backend_service" "ilb_backend_service_zonal" {
Step #2 - "tf plan": │ 
Step #2 - "tf plan": │ The argument "backend.0.group" is required, but no definition was found.
Step #2 - "tf plan": ╵

Yes, @edwardmedia it works with hardcoded data (as a work around) and it doesn't work properly (second terraform apply is required) when data source is used.

Yes, I checked data source returns - they are correct and if I run terraform apply second time there's no issue, as I stated before.

SehiiRohoza avatar Jan 28 '24 16:01 SehiiRohoza

@SehiiRohoza what do you see if you hardcode the data source google_compute_network_endpoint_group (removing count, etc) and provide group with below value in google_compute_region_backend_service?

group                 = data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal.id

edwardmedia avatar Jan 29 '24 16:01 edwardmedia

@SehiiRohoza what do you see if you hardcode the data source google_compute_network_endpoint_group (removing count, etc) and provide group with below value in google_compute_region_backend_service?

group                 = data.google_compute_network_endpoint_group.ilb_network_endpoint_group_zonal.id

@edwardmedia I tried the same syntax, but with count in my very first post and recently checked it again with Google cloud provider 5.X. Do you think that count can cause this issue? Let me provide you another example again, I have the same approach with External HTTPS LB: with data source and with count and it works. So I can plan and check it on some environment, but the chance that count has a some impact is pretty low.

SehiiRohoza avatar Jan 29 '24 21:01 SehiiRohoza

@SehiiRohoza I do see some issues when dynamic code is improperly used. But I can't comment on this particular use case. What we are concerned are individual resources and data sources. If you are able to repro an issue with hardcoded values, that could indicate it is possible a bug in the provider. We can help there.

In term of supporting different environments, have you looked at an approach where you create a module and then call the module with an environment value?

edwardmedia avatar Jan 30 '24 00:01 edwardmedia