ASM module behaves inconsistently with Autopilot clusters

[aablsk]

TL;DR

When using the ASM module with Autopilot clusters, terraform starts to detect drift upon subsequent runs, which then break the ASM deployment on the cluster (e.g. istio-validation containers failing continuously).

Expected behavior

ASM module works reliably with Autopilot clusters.

Observed behavior

While the initial deployment of ASM works, on subsequent runs terraform starts to detect drift and tries to fix resources which leads to a broken state (istio-validation containers failing continuously). I'm also unsure whether resource requests/limits are properly being set for Autopilot clusters.

Terraform Configuration

https://github.com/aablsk/bank-of-anthos/commit/8f55d9c635e81e6b10bd6411a52108911298bd62

Terraform Version

Terraform v1.2.9
on linux_amd64 

Cloud Console Shell default

Additional information

The new fleets API (https://cloud.google.com/service-mesh/docs/managed/automatic-management-with-fleet) provides a very convenient way to reliably enable ASM. While I believe that it allows for less configurability, the result is very consistent.

Example workaround I am currently using

module asm {
   source = "terraform-google-modules/gcloud/google"

    platform = "linux"

    create_cmd_entrypoint = "gcloud"
    create_cmd_body = "container fleet mesh update --management automatic --memberships ${google_gke_hub_membership.staging.membership_id} --project ${var.project_id}"
    destroy_cmd_entrypoint = "gcloud"
    destroy_cmd_body = "container fleet mesh update --management manual --memberships ${google_gke_hub_membership.staging.membership_id} --project ${var.project_id}"
}

[bharathkkb]

Thanks for the report @aablsk. What drift is detected on subsequent runs?

Regarding the workaround, this seems like we could actually use the provider resource now to manage this per https://github.com/hashicorp/terraform-provider-google/issues/11462. @Monkeyanator any objections?

[aablsk]

> What drift is detected on subsequent runs? @bharathkkb I'll have to reproduce to give more information. I expect to be able to find some time towards the end of the week and will report back latest next week.

[aablsk]

@bharathkkb sorry it took me longer than planned. I was not able to reproduce the reported drifting behaviour with most recent version of Autopilot.

[bharathkkb]

@aablsk Thanks for update. Feel free to reopen if it happens again to investigate.