Terraform providers fail to instantiate after multiple use

[doherty100]

To Reproduce

From the same cloudshell, apply multiple terraform configurations from different directories. After multiple terraform configurations have been applied, Terraform will start failing to initialize providers. This only happens on cloudshell. This can be consistently reproduced using #AzureSandbox.

This issue makes cloudshell unusable for complex Terraform scenarios.

Observed Behavior

Error: Failed to load plugin schemas

Error while loading schemas for plugin components: 2 problems:

- Failed to obtain provider schema: Could not load the schema for provider registry.terraform.io/hashicorp/azurerm: failed to instantiate provider
"registry.terraform.io/hashicorp/azurerm" to obtain schema: Unrecognized remote plugin message: 

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol..
- Failed to obtain provider schema: Could not load the schema for provider registry.terraform.io/hashicorp/random: failed to instantiate provider
"registry.terraform.io/hashicorp/random" to obtain schema: fork/exec .terraform/providers/registry.terraform.io/hashicorp/random/3.4.3/linux_amd64/terraform-provider-random_v3.4.3_x5:
exec format error..

Expected behavior

Terraform providers should instantiate without errors.

Is this specific to Cloud Shell?

Yes

Interface information

portal.azure.com

Additional context

https://github.com/hashicorp/terraform/issues/28354 https://github.com/hashicorp/terraform-provider-azurerm/issues/11543

[cpc-conden]

Im not running in CloudShell but I am intagrated with app.terraform.io and I get the same error when init and plan from my VSCode workstation.

Any work arounds?

`terraform init

Initializing the backend... Initializing modules...

• azurerm_storage_accounts in ......\implementation\storage-account-compound
• azurerm_storage_accounts.azurerm_storage_accounts in ......\modules\compound\terraform-azurerm-storage
• azurerm_storage_accounts.azurerm_storage_accounts.azurerm_private_endpoints in ......\modules\base\terraform-azurerm-private_endpoint
• azurerm_storage_accounts.azurerm_storage_accounts.azurerm_storage_account in ......\modules\base\terraform-azurerm-storage_account
• azurerm_storage_accounts.azurerm_storage_accounts.azurerm_storage_containers in ......\modules\base\terraform-azurerm-storage_container
• azurerm_storage_accounts.azurerm_storage_accounts.azurerm_storage_queues in ......\modules\base\terraform-azurerm-storage_queue- azurerm_storage_accounts.azurerm_storage_accounts.azurerm_storage_shares in ......\modules\base\terraform-azurerm-storage_share- azurerm_storage_accounts.azurerm_storage_accounts.azurerm_storage_tables in ......\modules\base\terraform-azurerm-storage_table Initializing provider plugins...
• terraform.io/builtin/terraform is built in to Terraform
• Finding hashicorp/azurerm versions matching "~> 3.50.0"...
• Installing hashicorp/azurerm v3.50.0...
• Installed hashicorp/azurerm v3.50.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.

If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. PS C:\Users.......> terraform plan ╷ │ Error: Failed to load plugin schemas │ │ Error while loading schemas for plugin components: Failed to obtain provider schema: Could not load the schema for provider │ registry.terraform.io/hashicorp/azurerm: failed to instantiate provider "registry.terraform.io/hashicorp/azurerm" to obtain │ schema: Unrecognized remote plugin message: │ │ This usually means that the plugin is either invalid or simply │ needs to be recompiled to support the latest protocol...`

[xXxH3LIOSxXx]

I have the exact same issue. Surprising that there are so few comments on this given how popular CloudShell is and how otherwise smooth the experience using it through VS Code is otherwise. Really hope this can get some attention!

One thing I did notice when this issue kicks in, is that successive runs of 'terraform validate' yield oscillating results. Perhaps this helps someone resolve the issue!

alan [ ~/clouddrive/theLab/Virtual Networks - Network Security Groups ]$ terraform validate
╷
│ Error: failed to read provider configuration schema for registry.terraform.io/hashicorp/azurerm: failed to instantiate provider "registry.terraform.io/hashicorp/azurerm" to obtain schema: Unrecognized remote plugin message:
│ 
│ This usually means that the plugin is either invalid or simply
│ needs to be recompiled to support the latest protocol.
│
│
╵
alan [ ~/clouddrive/theLab/Virtual Networks - Network Security Groups ]$ terraform validate
╷
│ Error: failed to read schema for azurerm_resource_group.rg-app1-sc-us-01 in registry.terraform.io/hashicorp/azurerm: failed to instantiate provider "registry.terraform.io/hashicorp/azurerm" to obtain schema: Unrecognized remote plugin message:     
│ 
│ This usually means that the plugin is either invalid or simply
│ needs to be recompiled to support the latest protocol.
│
│
╵
alan [ ~/clouddrive/theLab/Virtual Networks - Network Security Groups ]$ terraform validate
╷
│ Error: failed to read schema for azurerm_network_interface.NIC1 in registry.terraform.io/hashicorp/azurerm: failed to instantiate provider "registry.terraform.io/hashicorp/azurerm" to obtain schema: Unrecognized remote plugin message:
│ 
│ This usually means that the plugin is either invalid or simply
│ needs to be recompiled to support the latest protocol.
│
│ 
╵
alan [ ~/clouddrive/theLab/Virtual Networks - Network Security Groups ]$ terraform validate

The only other detail I can give is that when I first ran into this I wondered if it was somehow related to directory names because copying and renaming full directories of various .tf files, at times resolved the issue. Once resolved the issue stayed resolved. Once it starts, it stays.

[edyoung]

I don't have an exact diagnosis but it sounds like some local Terraform data may have gotten into a bad state.

This StackOverflow thread offers a couple of workarounds that may be useful:

https://stackoverflow.com/questions/70407525/terraform-gives-errors-failed-to-load-plugin-schemas