terraform-provider-azurerm
terraform-provider-azurerm copied to clipboard
BUG: windows app service with container is not getting created as expected
Is there an existing issue for this?
- [X] I have searched the existing issues
Community Note
- Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Azure windows app service with docker container is not getting created properly.
Expected: when we create terraform resource 'azurerm_windows_web_app' with 'application_stack' block having values such as docker_container_name, docker_container_tag, docker_container_registry. It should create app service with container, having an image as specified in application_stack' block.
But instead of creating app service with container features, it creates plane app service. Note- Creation happens properly in case of linux app service with container (azurerm_linux_web_app)
Here is the code which is taken from: windows-container
main.tf:
provider "azurerm" {
features {}
}
resource "azurerm_resource_group" "example" {
name = "${var.prefix}-resources"
location = var.location
}
resource "azurerm_service_plan" "example" {
name = "${var.prefix}-sp"
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
os_type = "Windows"
sku_name = "P1v3"
}
resource "azurerm_windows_web_app" "example" {
name = "${var.prefix}-container-example"
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
service_plan_id = azurerm_service_plan.example.id
app_settings = {
"WEBSITES_ENABLE_APP_SERVICE_STORAGE" = "false"
}
site_config {
application_stack {
docker_container_name = "jackofallops/azure-containerapps-python-acctest"
docker_container_tag = "v0.0.1"
}
}
}
variables.tf:
variable "prefix" {
type = string
description = "The prefix used for all resources in this example"
default = "mywebwincontainrc93"
}
variable "location" {
type = string
description = "The Azure location where all resources in this example should be created"
default = "West Europe"
}
Root Cause: While analyzing the different aspects of this issue, I exported the json template from azure portal, compared linux & windows json template and came to know that property: Kind is not getting set properly in case of windows app service hence the issue. In this case property: Kind is getting set as: "kind": "app" while it should be: "kind": "app,windows,container" While its getting set properly in case of linux app service with container i.e. "kind": "app,linux,container",
Refer image--> linux app service with container:
image--> windows app service with container, which is wrong:
Also it can be seen while destroying the resources too:
This issue blocks users to create an azure windows app service with container using terraform, which pulls image from specific container source, either docker or acr.
Terraform Version
1.1.6
AzureRM Provider Version
3.10.0
Affected Resource(s)/Data Source(s)
azurerm_windows_web_app
Terraform Configuration Files
resource "azurerm_windows_web_app" "example" {
name = "${var.prefix}-container-example"
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
service_plan_id = azurerm_service_plan.example.id
app_settings = {
"WEBSITES_ENABLE_APP_SERVICE_STORAGE" = "false"
}
site_config {
application_stack {
docker_container_name = "jackofallops/azure-containerapps-python-acctest"
docker_container_tag = "v0.0.1"
}
}
}
Debug Output/Panic Output
provided in screenshot
Expected Behaviour
should create azure windows app service container
Actual Behaviour
creating normal azure windows app service
Steps to Reproduce
- Clone code from : windows-container
- terraform init
- terraform plan
- terraform apply
- Once resources are created successfully, go to the app service from azure portal. In app service, select option 'Deployment Center' from the left side options menu
- It must show container details along with image details & tag. But its not doing so.
Important Factoids
No response
References
No response
@rohit93c Thanks for raising the issue, can you do me a favor which is to get the site config of this app by calling the below api?
https://management.azure.com/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Web/sites/xx/config/web?api-version=2021-02-01
@xiaxyi here is the site config:
{
"id": "/subscriptions/{mysubscription}/resourceGroups/{myrg}/providers/Microsoft.Web/sites/{mywebappcontainer}/config/web",
"name": "mywebappcontainer",
"type": "Microsoft.Web/sites/config",
"location": "West Europe",
"properties": {
"numberOfWorkers": 1,
"defaultDocuments": [
"Default.htm",
"Default.html",
"Default.asp",
"index.htm",
"index.html",
"iisstart.htm",
"default.aspx",
"index.php",
"hostingstart.html"
],
"netFrameworkVersion": "",
"phpVersion": "",
"pythonVersion": "",
"nodeVersion": "",
"powerShellVersion": "",
"linuxFxVersion": "",
"windowsFxVersion": "DOCKER|myacr.azurecr.io/nano:latest2",
"requestTracingEnabled": false,
"remoteDebuggingEnabled": false,
"remoteDebuggingVersion": "VS2019",
"httpLoggingEnabled": false,
"azureMonitorLogCategories": null,
"acrUseManagedIdentityCreds": true,
"acrUserManagedIdentityID": null,
"logsDirectorySizeLimit": 35,
"detailedErrorLoggingEnabled": false,
"publishingUsername": "$mywebappcontainer",
"publishingPassword": null,
"appSettings": null,
"metadata": null,
"connectionStrings": null,
"machineKey": null,
"handlerMappings": null,
"documentRoot": null,
"scmType": "None",
"use32BitWorkerProcess": false,
"webSocketsEnabled": false,
"alwaysOn": true,
"javaVersion": null,
"javaContainer": null,
"javaContainerVersion": null,
"appCommandLine": "",
"managedPipelineMode": "Integrated",
"virtualApplications": [
{
"virtualPath": "/",
"physicalPath": "site\\wwwroot",
"preloadEnabled": true,
"virtualDirectories": null
}
],
"winAuthAdminState": 0,
"winAuthTenantState": 0,
"customAppPoolIdentityAdminState": false,
"customAppPoolIdentityTenantState": false,
"runtimeADUser": null,
"runtimeADUserPassword": null,
"loadBalancing": "LeastRequests",
"routingRules": [],
"experiments": {
"rampUpRules": []
},
"limits": null,
"autoHealEnabled": false,
"autoHealRules": null,
"tracingOptions": null,
"vnetName": "",
"vnetRouteAllEnabled": false,
"vnetPrivatePortsCount": 0,
"publicNetworkAccess": null,
"siteAuthEnabled": false,
"siteAuthSettings": {
"enabled": null,
"unauthenticatedClientAction": null,
"tokenStoreEnabled": null,
"allowedExternalRedirectUrls": null,
"defaultProvider": null,
"clientId": null,
"clientSecret": null,
"clientSecretSettingName": null,
"clientSecretCertificateThumbprint": null,
"issuer": null,
"allowedAudiences": null,
"additionalLoginParams": null,
"isAadAutoProvisioned": false,
"aadClaimsAuthorization": null,
"googleClientId": null,
"googleClientSecret": null,
"googleClientSecretSettingName": null,
"googleOAuthScopes": null,
"facebookAppId": null,
"facebookAppSecret": null,
"facebookAppSecretSettingName": null,
"facebookOAuthScopes": null,
"gitHubClientId": null,
"gitHubClientSecret": null,
"gitHubClientSecretSettingName": null,
"gitHubOAuthScopes": null,
"twitterConsumerKey": null,
"twitterConsumerSecret": null,
"twitterConsumerSecretSettingName": null,
"microsoftAccountClientId": null,
"microsoftAccountClientSecret": null,
"microsoftAccountClientSecretSettingName": null,
"microsoftAccountOAuthScopes": null,
"configVersion": null
},
"cors": null,
"push": null,
"apiDefinition": null,
"apiManagementConfig": null,
"autoSwapSlotName": null,
"localMySqlEnabled": false,
"managedServiceIdentityId": 42009,
"xManagedServiceIdentityId": null,
"keyVaultReferenceIdentity": null,
"ipSecurityRestrictions": [
{
"ipAddress": "Any",
"action": "Allow",
"priority": 1,
"name": "Allow all",
"description": "Allow all access"
}
],
"scmIpSecurityRestrictions": [
{
"ipAddress": "Any",
"action": "Allow",
"priority": 1,
"name": "Allow all",
"description": "Allow all access"
}
],
"scmIpSecurityRestrictionsUseMain": false,
"http20Enabled": false,
"minTlsVersion": "1.2",
"minTlsCipherSuite": null,
"supportedTlsCipherSuites": null,
"scmMinTlsVersion": "1.2",
"ftpsState": "Disabled",
"preWarmedInstanceCount": 0,
"functionAppScaleLimit": 0,
"healthCheckPath": null,
"fileChangeAuditEnabled": false,
"functionsRuntimeScaleMonitoringEnabled": false,
"websiteTimeZone": null,
"minimumElasticInstanceCount": 0,
"azureStorageAccounts": {},
"http20ProxyFlag": 0,
"sitePort": null,
"antivirusScanEnabled": false,
"storageType": "StorageVolume"
}
}
Also FYI.. The deployment center option doesn't show properties related to container due to this issue:
Thanks @rohit93c , can you help me to try to set the .NETFrameworkVersion to "v4.0" to see if the app can be started as expected?
@xiaxyi Do you want me set dotnet_version = "v4.0" inside 'application_stack' block?
Please don't use terraform, instead, can you try using PUT method to set the property "netFrameworkVersion": "v4.0"
in the api?
@xiaxyi getting 400 error: { "error": { "code": "InvalidRequestContent", "message": "The request content was invalid and could not be deserialized: 'Could not find member 'properties.netFrameworkVersion' on object of type 'ResourceDefinition'. Path '['properties.netFrameworkVersion']', line 2, position 41.'." } }
are you using the api https://management.azure.com/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Web/sites/xx/config/web?api-version=2021-02-01
with the PUT method and with the .NETFrameworkVersion sets to v4.0 in the body?
@xiaxyi successfully updated the said property:
snap from azure export template:
but still the issue persists
Thanks @rohit93c , I noticed that you are using "Windows" as the os_type in app_service_plan resource:
resource "azurerm_service_plan" "example" {
name = "${var.prefix}-sp"
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
os_type = "Windows"
sku_name = "P1v3"
}
Can you try WindowsContainer
?
@xiaxyi
Thanks @rohit93c , I noticed that you are using "Windows" as the os_type in app_service_plan resource:
resource "azurerm_service_plan" "example" { name = "${var.prefix}-sp" location = azurerm_resource_group.example.location resource_group_name = azurerm_resource_group.example.name os_type = "Windows" sku_name = "P1v3" }
Can you try
WindowsContainer
?
@xiaxyi Tried with : os_type = "WindowsContainer"
terraform apply -auto-approve
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# module.appplan.azurerm_service_plan.az_app_plan will be created
+ resource "azurerm_service_plan" "az_app_plan" {
+ id = (known after apply)
+ kind = (known after apply)
+ location = "westeurope"
+ maximum_elastic_worker_count = (known after apply)
+ name = "mywebplan"
+ os_type = "WindowsContainer"
+ per_site_scaling_enabled = false
+ reserved = (known after apply)
+ resource_group_name = "myrg"
+ sku_name = "P1v3"
+ worker_count = (known after apply)
}
# module.rg.azurerm_resource_group.az_rg will be created
+ resource "azurerm_resource_group" "az_rg" {
+ id = (known after apply)
+ location = "westeurope"
+ name = "myrg"
}
# module.webapp_container_win.azurerm_role_assignment.az_role_assign will be created
+ resource "azurerm_role_assignment" "az_role_assign" {
+ id = (known after apply)
+ name = (known after apply)
+ principal_id = (known after apply)
+ principal_type = (known after apply)
+ role_definition_id = (known after apply)
+ role_definition_name = "AcrPull"
+ scope = "/subscriptions/mysubscription/resourceGroups/myacrrg/providers/Microsoft.ContainerRegistry/registries/myacr"
+ skip_service_principal_aad_check = true
}
# module.webapp_container_win.azurerm_windows_web_app.az_win_web_app will be created
+ resource "azurerm_windows_web_app" "az_win_web_app" {
+ app_settings = {
+ "WEBSITES_ENABLE_APP_SERVICE_STORAGE" = "false"
}
+ client_affinity_enabled = false
+ client_certificate_enabled = false
+ client_certificate_mode = "Required"
+ custom_domain_verification_id = (sensitive value)
+ default_hostname = (known after apply)
+ enabled = true
+ https_only = true
+ id = (known after apply)
+ key_vault_reference_identity_id = (known after apply)
+ kind = (known after apply)
+ location = "westeurope"
+ name = "mywebcontainer"
+ outbound_ip_address_list = (known after apply)
+ outbound_ip_addresses = (known after apply)
+ possible_outbound_ip_address_list = (known after apply)
+ possible_outbound_ip_addresses = (known after apply)
+ resource_group_name = "myrg"
+ service_plan_id = (known after apply)
+ site_credential = (known after apply)
+ zip_deploy_file = (known after apply)
+ auth_settings {
+ additional_login_parameters = (known after apply)
+ allowed_external_redirect_urls = (known after apply)
+ default_provider = (known after apply)
+ enabled = (known after apply)
+ issuer = (known after apply)
+ runtime_version = (known after apply)
+ token_refresh_extension_hours = (known after apply)
+ token_store_enabled = (known after apply)
+ unauthenticated_client_action = (known after apply)
+ active_directory {
+ allowed_audiences = (known after apply)
+ client_id = (known after apply)
+ client_secret = (sensitive value)
+ client_secret_setting_name = (known after apply)
}
+ facebook {
+ app_id = (known after apply)
+ app_secret = (sensitive value)
+ app_secret_setting_name = (known after apply)
+ oauth_scopes = (known after apply)
}
+ github {
+ client_id = (known after apply)
+ client_secret = (sensitive value)
+ client_secret_setting_name = (known after apply)
+ oauth_scopes = (known after apply)
}
+ google {
+ client_id = (known after apply)
+ client_secret = (sensitive value)
+ client_secret_setting_name = (known after apply)
+ oauth_scopes = (known after apply)
}
+ microsoft {
+ client_id = (known after apply)
+ client_secret = (sensitive value)
+ client_secret_setting_name = (known after apply)
+ oauth_scopes = (known after apply)
}
+ twitter {
+ consumer_key = (known after apply)
+ consumer_secret = (sensitive value)
+ consumer_secret_setting_name = (known after apply)
}
}
+ identity {
+ principal_id = (known after apply)
+ tenant_id = (known after apply)
+ type = "SystemAssigned"
}
+ site_config {
+ always_on = true
+ auto_heal_enabled = false
+ container_registry_use_managed_identity = true
+ default_documents = (known after apply)
+ detailed_error_logging_enabled = (known after apply)
+ ftps_state = "Disabled"
+ health_check_eviction_time_in_min = (known after apply)
+ http2_enabled = false
+ ip_restriction = (known after apply)
+ linux_fx_version = (known after apply)
+ load_balancing_mode = "LeastRequests"
+ local_mysql_enabled = false
+ managed_pipeline_mode = "Integrated"
+ minimum_tls_version = "1.2"
+ remote_debugging_enabled = false
+ remote_debugging_version = (known after apply)
+ scm_ip_restriction = (known after apply)
+ scm_minimum_tls_version = "1.2"
+ scm_type = (known after apply)
+ scm_use_main_ip_restriction = false
+ use_32_bit_worker = false
+ vnet_route_all_enabled = false
+ websockets_enabled = false
+ windows_fx_version = (known after apply)
+ worker_count = (known after apply)
+ application_stack {
+ docker_container_name = "nano"
+ docker_container_registry = "myacr.azurecr.io"
+ docker_container_tag = "latest"
}
}
}
Getting error: access denied to azure container registry
Plan: 4 to add, 0 to change, 0 to destroy.
module.rg.azurerm_resource_group.az_rg: Creating...
module.rg.azurerm_resource_group.az_rg: Creation complete after 4s [id=/subscriptions/mysubscription/resourceGroups/myrg]
module.appplan.azurerm_service_plan.az_app_plan: Creating...
module.appplan.azurerm_service_plan.az_app_plan: Still creating... [10s elapsed]
module.appplan.azurerm_service_plan.az_app_plan: Creation complete after 14s [id=/subscriptions/mysubscription/resourceGroups/myrg/providers/Microsoft.Web/serverfarms/mywebplan]
module.webapp_container_win.azurerm_windows_web_app.az_win_web_app: Creating...
module.webapp_container_win.azurerm_windows_web_app.az_win_web_app: Still creating... [10s elapsed]
module.webapp_container_win.azurerm_windows_web_app.az_win_web_app: Still creating... [20s elapsed]
╷
│ Error: creating Windows Web App: (Site Name "mywebcontainer" / Resource Group "myrg"): web.AppsClient#CreateOrUpdate: Failure sending request: StatusCode=401 -- Original Error: Code="Unauthorized" Message="Access is denied. Not authorized. https://myacr.azurecr.io/" Details=[{"Message":"Access is denied. Not authorized. https://myacr.azurecr.io/"},{"Code":"Unauthorized"},{"ErrorEntity":{"Code":"Unauthorized","ExtendedCode":"01001","Message":"Access is denied. Not authorized. https://myacr.azurecr.io/","MessageTemplate":"Access is denied.","Parameters":[]}}]
│
│ with module.webapp_container_win.azurerm_windows_web_app.az_win_web_app,
│ on ..\azr-iac-module-webcontainer-win\main.tf line 16, in resource "azurerm_windows_web_app" "az_win_web_app":
│ 16: resource "azurerm_windows_web_app" "az_win_web_app" {
│
│ creating Windows Web App: (Site Name "mywebcontainer" / Resource Group "myrg"): web.AppsClient#CreateOrUpdate: Failure sending request:
│ StatusCode=401 -- Original Error: Code="Unauthorized" Message="Access is denied. Not authorized. https://myacr.azurecr.io/"
│ Details=[{"Message":"Access is denied. Not authorized.
│ https://myacr.azurecr.io/"},{"Code":"Unauthorized"},{"ErrorEntity":{"Code":"Unauthorized","ExtendedCode":"01001","Message":"Access is denied. Not
│ authorized. https://myacr.azurecr.io/","MessageTemplate":"Access is denied.","Parameters":[]}}]
@xiaxyi The above same error i get, when i try to deploy web container using ARM template (in case through terraform as well as through azure portal-template deployment.)
in case of ARM template i tried to include "kind": "app,windows,container" as this seems to be the issues in case when we tried to create windows webapp container through terraform
sorry @rohit93c I'm kind of confused here.
can you do me a favor to perform a test to create a new app service instead of making changes to the existing one.
steps:
- Creating the app service with the os_type sets to "WindowsContainer"
- After the web app is created, can you try to update the .NETFrameworkVersion property to v4.0 and see if the container app can be started as expected?
Thanks~
@xiaxyi i am trying to create windows app service container (azurerm_windows_web_app). This web app will be pulling an image from azure container registry using code block:
application_stack {
docker_container_name = "nano"
docker_container_registry = "myacr.azurecr.io"
docker_container_tag = "latest"
}
For webapp to be able to pull images from container registry using managed identity, i am using role assignment (azurerm_role_assignment). Following code will run once azurerm_windows_web_app creation is completed.
resource "azurerm_role_assignment" "az_role_assign" {
principal_id = azurerm_windows_web_app.az_win_web_app.identity[0].principal_id
role_definition_name = "AcrPull"
scope = data.azurerm_container_registry.az_acr.id
skip_service_principal_aad_check = true
}
When i keep os_type = "Windows" for app service plan (azurerm_service_plan), it created simple web app & not the webapp container one.
As suggested by you in one of the comments above, when i keep os_type = "WindowsContainer" for app service plan, web app (azurerm_windows_web_app) creation fails with an error:
Error: creating Windows Web App: (Site Name "mywebcontainer" / Resource Group "myrg"): web.AppsClient#CreateOrUpdate: Failure sending request: StatusCode=401 -- Original Error: Code="Unauthorized" Message="Access is denied. Not authorized. https://myacr.azurecr.io/" Details=[{"Message":"Access is denied. Not authorized. https://myacr.azurecr.io/"},{"Code":"Unauthorized"},{"ErrorEntity":{"Code":"Unauthorized","ExtendedCode":"01001","Message":"Access is denied. Not authorized. https://myacr.azurecr.io/","MessageTemplate":"Access is denied.","Parameters":[]}}]
│
│ with module.webapp_container_win.azurerm_windows_web_app.az_win_web_app,
│ on ..\azr-iac-module-webcontainer-win\main.tf line 16, in resource "azurerm_windows_web_app" "az_win_web_app":
│ 16: resource "azurerm_windows_web_app" "az_win_web_app" {
│
│ creating Windows Web App: (Site Name "mywebcontainer" / Resource Group "myrg"): web.AppsClient#CreateOrUpdate: Failure sending request:
│ StatusCode=401 -- Original Error: Code="Unauthorized" Message="Access is denied. Not authorized. https://myacr.azurecr.io/"
│ Details=[{"Message":"Access is denied. Not authorized.
│ https://myacr.azurecr.io/"},{"Code":"Unauthorized"},{"ErrorEntity":{"Code":"Unauthorized","ExtendedCode":"01001","Message":"Access is denied. Not
│ authorized. https://myacr.azurecr.io/","MessageTemplate":"Access is denied.","Parameters":[]}}]
@rohit93c If my understanding is correct, are you suggesting that you can't use managed identity to access acr if the os_type is set to windowsContainer?
@rohit93c If my understanding is correct, are you suggesting that you can't use managed identity to access acr if the os_type is set to windowsContainer?
@xiaxyi yes seems to be so. It gives error: "Access is denied. Not authorized" when os_type is set to windowsContainer. And web app container creation fails
Thanks @rohit93c let me try from my side
Thanks @rohit93c let me try from my side
Sure. Thanks for the help & suggestions. Requesting you to please keep me updated here on this issue, as it seems to be a kind of blocker, which blocks us to create windows web app with container.
@xiaxyi hi, any solution/update on the above issue?
This is also happening to me currently, but only when I try to bring a pre-created app service plan with OS type = windows.
One thing I find confusing about this is that there seems to be no way to create an App Service Plan via Azure Portal with os_type = "WindowsContainer". Am I missing something?
I know that I can do this other ways, e.g. Azure CLI or Terraform.
Even if I do create an app service plan with value "WindowsContainer" and try to use that to create the Web App for Containers, I then run into this other issue: https://github.com/hashicorp/terraform-provider-azurerm/issues/19173
Any Updates on this?
At this moment, I have to create the Web App manually and then import into my Terraform Configuration.
Thanks in advance.
Not sure if this issue is fixed or not. But the workaround i used for it is to create a plain webapp by keeping application stack configurations blank. And then setting docker container image & tag details by firing az cli commands (null_resource can be used to achieve this).
Create a webapp
resource "azurerm_windows_web_app" "example" {
name = "${var.prefix}-container-example"
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
service_plan_id = azurerm_service_plan.example.id
app_settings = {
"WEBSITES_ENABLE_APP_SERVICE_STORAGE" = "false"
}
site_config {
application_stack {
}
}
}
Run local-exec to set image
resource "null_resource" "az_set_container_image" {
depends_on = [azurerm_windows_web_app.example]
provisioner "local-exec" {
command = "az webapp config container set -g azurerm_resource_group.example.name -n azurerm_windows_web_app.example.name --docker-custom-image-name myimage --docker-registry-server-url myacr.azurecr.io/myimage:latest"
}
}