error message with typos, unable to expand variable group

[RolfMoleman]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and Azure DevOps Provider) Version

Terraform v1.6.3 on windows_amd64

• provider registry.terraform.io/hashicorp/azuread v2.45.0
• provider registry.terraform.io/hashicorp/azurerm v3.79.0
• provider registry.terraform.io/hashicorp/random v3.5.1
• provider registry.terraform.io/hashicorp/time v0.9.1
• provider registry.terraform.io/microsoft/azuredevops v0.10.0

Affected Resource(s)

• azuredevops_variable_group

Terraform Configuration Files

## Service Connection
resource "azuredevops_serviceendpoint_azurerm" "bootstrap" {
  project_id            = data.azuredevops_project.bootstrap.id
  service_endpoint_name = join("", [data.azurerm_subscription.bootstrap.display_name, " - ", title(var.environment_tag), " environment"])
  description           = join("", [data.azurerm_subscription.bootstrap.display_name, " subscription service connection for ", title(var.environment_tag), " environment that will expire on ", time_offset.spn_password_expiry.day, "/", time_offset.spn_password_expiry.month, "/", time_offset.spn_password_expiry.year])
  credentials {
    serviceprincipalid  = azuread_service_principal.bootstrap.client_id
    serviceprincipalkey = azuread_service_principal_password.bootstrap.value
  }
  azurerm_spn_tenantid      = data.azuread_client_config.bootstrap.tenant_id
  azurerm_subscription_id   = data.azurerm_client_config.bootstrap.subscription_id
  azurerm_subscription_name = data.azurerm_subscription.bootstrap.display_name
  depends_on = [
    azuread_application.bootstrap
  ]
}

## Grant permission to use service connection
resource "azuredevops_pipeline_authorization" "service_endpoint" {
  project_id  = data.azuredevops_project.bootstrap.id
  resource_id = azuredevops_serviceendpoint_azurerm.bootstrap.id
  type        = "endpoint"
  depends_on = [
    azuredevops_serviceendpoint_azurerm.bootstrap
  ]
}

## Variable group linked to keyvault
#If Variable Group is linked to a Key Vault, only top 500 secrets will be read by default. Key Vault does not support filter the secret by name, we can only read the secrets and do filter in Terraform.
resource "azuredevops_variable_group" "bootstrap" {
  project_id   = data.azuredevops_project.bootstrap.id
  name         = join("-", ["tfvars", var.environment_tag])
  description  = join(" ", [var.environment_tag, "environment Terraform variables to be used for infrastructure as code pipelines. the keyvault is called", azurerm_key_vault.bootstrap.name])
  allow_access = true
  depends_on = [
    azuread_application.bootstrap, azuredevops_serviceendpoint_azurerm.bootstrap, azuredevops_pipeline_authorization.service_endpoint
  ]
  key_vault {
    name                = azurerm_key_vault.bootstrap.name
    service_endpoint_id = azuredevops_serviceendpoint_azurerm.bootstrap.id
  }

  variable {
    name = "access-key"
  }

  variable {
    name = "client-id"
  }

  variable {
    name = "client-secret"
  }

  variable {
    name = "container-name"
  }

  variable {
    name = "default-tags"
  }

  variable {
    name = "resource-group"
  }

  variable {
    name = "subscription-id"
  }

  variable {
    name = "storage-account"
  }

  variable {
    name = "tenant-id"
  }

}

## Grant permission to use variable group
resource "azuredevops_pipeline_authorization" "variable_group" {
  project_id  = data.azuredevops_project.bootstrap.id
  resource_id = azuredevops_variable_group.bootstrap.id
  type        = "variablegroup"
  depends_on = [
    azuredevops_serviceendpoint_azurerm.bootstrap, azuredevops_pipeline_authorization.service_endpoint
  ]
}

Expected Behavior

variable group should have expanded and been populated by the linked key vault or should it have errored then the error should have been:

│ Error: Expanding variable group resource data: Failed to get the Azure Key vault. Error: ( code: badRequest, message: Failed to obtain the Json Web Token(JWT) using service principal client ID. Exception message: A configuration issue is preventing authentication - check the error message from the server for details. You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details. Original exception: AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '~redacted~'. Trace ID: 18d3f700-8059-4688-9a7b-c8d018e26a00 Correlation ID: 8c00024e-4ac3-4bdc-af1b-75e71f9438ff Timestamp: 2023-11-07 16:12:38Z ) │ │ with azuredevops_variable_group.bootstrap, │ on devops.tf line 31, in resource "azuredevops_variable_group" "bootstrap": │ 31: resource "azuredevops_variable_group" "bootstrap" { │ ╵ 2023-11-07T16:12:39.050Z [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info 2023-11-07T16:12:39.050Z [TRACE] statemgr.Filesystem: unlocked by closing terraform.tfstate 2023-11-07T16:12:39.052Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF" 2023-11-07T16:12:39.052Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF" 2023-11-07T16:12:39.074Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/microsoft/azuredevops/0.10.0/windows_amd64/terraform-provider-azuredevops_v0.10.0 pid=11744 2023-11-07T16:12:39.075Z [DEBUG] provider: plugin exited 2023-11-07T16:12:39.090Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.79.0/windows_amd64/terraform-provider-azurerm_v3.79.0_x5.exe pid=37220 2023-11-07T16:12:39.091Z [DEBUG] provider: plugin exited

Actual Behavior

│ Error: Expanding variable group resource data: Failed to get the Azure Key valut. Erroe: ( code: badRequest, messge: Failed to obtain the Json Web Token(JWT) using service principal client ID. Exception message: A configuration issue is preventing authentication - check the error message from the server for details. You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details. Original exception: AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '~redacted~'. Trace ID: 18d3f700-8059-4688-9a7b-c8d018e26a00 Correlation ID: 8c00024e-4ac3-4bdc-af1b-75e71f9438ff Timestamp: 2023-11-07 16:12:38Z ) │ │ with azuredevops_variable_group.bootstrap, │ on devops.tf line 31, in resource "azuredevops_variable_group" "bootstrap": │ 31: resource "azuredevops_variable_group" "bootstrap" { │ ╵ 2023-11-07T16:12:39.050Z [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info 2023-11-07T16:12:39.050Z [TRACE] statemgr.Filesystem: unlocked by closing terraform.tfstate 2023-11-07T16:12:39.052Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF" 2023-11-07T16:12:39.052Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF" 2023-11-07T16:12:39.074Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/microsoft/azuredevops/0.10.0/windows_amd64/terraform-provider-azuredevops_v0.10.0 pid=11744 2023-11-07T16:12:39.075Z [DEBUG] provider: plugin exited 2023-11-07T16:12:39.090Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.79.0/windows_amd64/terraform-provider-azurerm_v3.79.0_x5.exe pid=37220 2023-11-07T16:12:39.091Z [DEBUG] provider: plugin exited

Steps to Reproduce

1. terraform plan --out=plan_local --var="cag_division=BCA" --var="environment_tag=dev" --var="azdo_project_name=~redacted~" --var="azdo_pat=~redacted~"
2. terraform apply --auto-approve "plan_local"

Important Factoids

References

• #0000

[xuzhang3]

@RolfMoleman Can you verify the SPN is correct. From the error log the SPN secret is invalid: Invalid client secret provided

[RolfMoleman]

Hi,

The SPN secret is valid.

Found the issue to be due to recent changes in the AAD provider requiring serviceprincipalid to be set to azuread_application.bootstrap.client_id rather than azuread_service_principal.bootstrap.id and serviceprinicpalkey to be set to azuread_application_password.bootstrap.value as opposed to azuread_service_principal_password.bootstrap.value

[rdalbuquerque]

I think this might actually be related to #825 where keyvault-linked variable groups fails to expand if the aad application password backing the service connection is freshly created or changed

there is an opened PR (#865) to fix this issue but looks like the problem lies beyond the reach of the API or this provider

[RolfMoleman]

That issue still exists in the provider, and i did encounter that before this.

The core issues for me are:

• Errors in the error message
• Inconsistency between the azuread, azurerm and azuredevops providers about id names. I.e is it client Id,app id, id or object id and why isn't it explicitly stated in the docs and in the parameter name.
• bizarre behaviour where retrieving the password from the app reg works fine but not from the spn

[RolfMoleman]

I think this might actually be related to #825 where keyvault-linked variable groups fails to expand if the aad application password backing the service connection is freshly created or changed

there is an opened PR (#865) to fix this issue but looks like the problem lies beyond the reach of the API or this provider

@rdalbuquerque It is related to this yes.

However, this could be overcome by a documentation change, variable/parameter name consistency and a correctly worded error message

[RolfMoleman]

@rdalbuquerque

I have managed a work around of this at the moment using the time provider but the error message from the azuredevops probider still requires correcting and the input parameters still require consistency.

Workaround:

Hcl

resource "time_sleep" "wait_for_rbac" {
  depends_on = [
    azurerm_role_assignment.BCA_Cloud_DevOps_Engineers,
    azurerm_role_assignment.BCA_Cloud_DevOps_Engineers,
    azurerm_role_assignment.role_based_access_control_administrator,
    azurerm_role_assignment.key_vault_administrator,
  azurerm_role_assignment.platform_automation_team]

  create_duration = "2m" # 2 minutes
}

## Service Connection
resource "azuredevops_serviceendpoint_azurerm" "bootstrap" {
  project_id            = data.azuredevops_project.bootstrap.id
  service_endpoint_name = join("", [data.azurerm_subscription.bootstrap.display_name, " - ", title(var.environment_tag), " environment"])
  description           = join("", [data.azurerm_subscription.bootstrap.display_name, " subscription service connection for ", title(var.environment_tag), " environment that will expire on ", time_offset.spn_password_expiry.day, "/", time_offset.spn_password_expiry.month, "/", time_offset.spn_password_expiry.year])
  credentials {
    serviceprincipalid  = azuread_service_principal.bootstrap.client_id
    serviceprincipalkey = azuread_service_principal_password.bootstrap.value
  }
  azurerm_spn_tenantid      = data.azuread_client_config.bootstrap.tenant_id
  azurerm_subscription_id   = data.azurerm_client_config.bootstrap.subscription_id
  azurerm_subscription_name = data.azurerm_subscription.bootstrap.display_name
  depends_on = [
    azuread_application.bootstrap,
    time_sleep.wait_for_rbac
  ]
}

## Grant permission to use service connection
resource "azuredevops_pipeline_authorization" "service_endpoint" {
  project_id  = data.azuredevops_project.bootstrap.id
  resource_id = azuredevops_serviceendpoint_azurerm.bootstrap.id
  type        = "endpoint"
  depends_on = [
    azuredevops_serviceendpoint_azurerm.bootstrap,
    time_sleep.wait_for_rbac
  ]
}

## Variable group linked to keyvault
#If Variable Group is linked to a Key Vault, only top 500 secrets will be read by default. Key Vault does not support filter the secret by name, we can only read the secrets and do filter in Terraform.
resource "azuredevops_variable_group" "bootstrap" {
  project_id   = data.azuredevops_project.bootstrap.id
  name         = join("-", ["tfvars", var.environment_tag])
  description  = join(" ", [var.environment_tag, "environment Terraform variables to be used for infrastructure as code pipelines. the keyvault is called", azurerm_key_vault.bootstrap.name])
  allow_access = true
  depends_on = [
    azuread_application.bootstrap,
    azuredevops_serviceendpoint_azurerm.bootstrap,
    azuredevops_pipeline_authorization.service_endpoint,
    time_sleep.wait_for_rbac
  ]
  key_vault {
    name                = azurerm_key_vault.bootstrap.name
    service_endpoint_id = azuredevops_serviceendpoint_azurerm.bootstrap.id
  }

  variable {
    name = "access-key"
  }

  variable {
    name = "client-id"
  }

  variable {
    name = "client-secret"
  }

  variable {
    name = "container-name"
  }

  variable {
    name = "default-tags"
  }

  variable {
    name = "resource-group"
  }

  variable {
    name = "subscription-id"
  }

  variable {
    name = "storage-account"
  }

  variable {
    name = "tenant-id"
  }

}

## Grant permission to use variable group
resource "azuredevops_pipeline_authorization" "variable_group" {
  project_id  = data.azuredevops_project.bootstrap.id
  resource_id = azuredevops_variable_group.bootstrap.id
  type        = "variablegroup"
  depends_on = [
    azuredevops_serviceendpoint_azurerm.bootstrap,
    azuredevops_pipeline_authorization.service_endpoint,
    time_sleep.wait_for_rbac
  ]
}