Unexpected Error from plugin.(*GRPCProvider).UpgradeResourceState

[james1miller93]

We're receiving intermittent errors from the provider when running terraform plan against crd resources in a cluster which are managed using the provider.

We only seem to see these errors in <10% of plan operations and they seem to occur on resources which are not being modified in the plan.

• terraform v1.2.4
• gavinbunney/kubectl v1.14.0

Plan error

│ Error: Plugin error
│
│   with module.aks.module.core_config.kubectl_manifest.kube_prometheus_stack_crds["alertmanagers.monitoring.coreos.com.yaml"],
│   on modules/terraform-azurerm-aks/modules/core-config/main.tf line 19, in resource "kubectl_manifest" "kube_prometheus_stack_crds":
│   19: resource "kubectl_manifest" "kube_prometheus_stack_crds" {
│
│ The plugin returned an unexpected error from
│ plugin.(*GRPCProvider).UpgradeResourceState: rpc error: code = Internal
│ desc = unexpected EOF

Potentially relevant debug log data

2022-06-30T11:38:55.416+0100 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/gavinbunney/kubectl/1.14.0/darwin_amd64/terraform-provider-kubectl_v1.14.0 args=[.terraform/providers/registry.terraform.io/gavinbunney/kubectl/1.14.0/darwin_amd64/terraform-provider-kubectl_v1.14.0]
2022-06-30T11:38:55.422+0100 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/gavinbunney/kubectl/1.14.0/darwin_amd64/terraform-provider-kubectl_v1.14.0 pid=72521
2022-06-30T11:38:55.423+0100 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/gavinbunney/kubectl/1.14.0/darwin_amd64/terraform-provider-kubectl_v1.14.0
2022-06-30T11:38:55.466+0100 [INFO]  provider.terraform-provider-kubectl_v1.14.0: configuring server automatic mTLS: timestamp=2022-06-30T11:38:55.465+0100
2022-06-30T11:38:55.505+0100 [DEBUG] provider: using plugin: version=5
2022-06-30T11:38:55.505+0100 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: plugin address: address=/var/folders/_x/s5h94k015mz8xvd8dn3_gpp9bq4p14/T/plugin684760236 network=unix timestamp=2022-06-30T11:38:55.504+0100
2022-06-30T11:38:55.541+0100 [TRACE] GRPCProvider: GetProviderSchema
2022-06-30T11:38:55.541+0100 [TRACE] provider.stdio: waiting for stdio data
2022-06-30T11:38:55.543+0100 [TRACE] GRPCProvider: Close
2022-06-30T11:38:55.544+0100 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-06-30T11:38:55.546+0100 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/gavinbunney/kubectl/1.14.0/darwin_amd64/terraform-provider-kubectl_v1.14.0 pid=72521
2022-06-30T11:38:55.546+0100 [DEBUG] provider: plugin exited

2022-06-30T11:39:22.162+0100 [ERROR] plugin.(*GRPCProvider).UpgradeResourceState: error="rpc error: code = Internal desc = unexpected EOF"
2022-06-30T11:39:22.162+0100 [ERROR] vertex "module.aks.module.core_config.kubectl_manifest.kube_prometheus_stack_crds[\"alertmanagers.monitoring.coreos.com.yaml\"]" error: Plugin error
2022-06-30T11:39:22.162+0100 [TRACE] vertex "module.aks.module.core_config.kubectl_manifest.kube_prometheus_stack_crds[\"alertmanagers.monitoring.coreos.com.yaml\"]": visit complete, with errors

I have a full TRACE log for a failed job, so can provide more log data if needed.

[mhulscher]

Wanted to mention that we are experiencing the same issue and it is introducing flakiness into our pipelines :(

[aydosman]

@gavinbunney, I'm no go expert but could this not be solved by putting a retry & back off in the ReadContext

[ryanmcafee]

We're also impacted by this issue and it's causing intermittent pipeline failures for us like @mhulscher.