[Issue] Can't deploy environment that includes a PostGreSQL flexible server in eastus2

[pamelafox]

trafficstars

Output from azd version

azd version 0.3.0-beta.2 (commit 91a2c014887ebda63a0af2e623278f8ac40ce376

Output from az version

{
  "azure-cli": "2.40.0",
  "azure-cli-core": "2.40.0",
  "azure-cli-telemetry": "1.0.8",
  "extensions": {
    "containerapp": "0.3.11",
    "containerapp-compose": "0.2.2"
  }
} 

Describe the bug

When I attempt to employ the django sample app in eastus2, it fails with an error like:

Created Resource group: django-sample10-rg
Created App Service plan: django-sample10-vsr7nhivcwl7g-service-plan
Created Log Analytics workspace: django-sample10-vsr7nhivcwl7g-workspace
Created Web App: django-sample10-vsr7nhivcwl7g-app-service
Error: deploying infrastructure: error deploying infrastructure: failed deploying: failed running az deployment sub create: 

Deployment Error Details:
InternalServerError: An unexpected error occured while processing the request. Tracking ID: '96a679dd-cdf8-4f8a-b1b3-5a54d6792569'

Screenshot from Portal of failed deployment:

I did 11 rounds of deployment and tracked success based on various factors, and location was the only one affecting success/failure. As far as I can tell, any attempt to deploy to eastus2 will fail like that.

To Reproduce

1. Git clone https://github.com/pamelafox/msdocs-django-postgresql-sample-app-azd
2. Create an environment in eastus2
3. See it fail
4. Change environment to eastus
5. See it succeed!

Additional context

I encountered the same issue when deploying to eastus2 with another app that has a PostGreSQL database, clicking the "Deploy to Azure" button on https://github.com/pamelafox/babybuddy-azure So, this isn't specific to azd CLI, but it does affect users of it, so perhaps you can help redirect this issue to the underlying service.

[tonybaloney]

Was there any additional details in the Operation Details?

[pamelafox]

Not really- the status is "Conflict" and here's the status message:

{
    "status": "Failed",
    "error": {
        "code": "InternalServerError",
        "message": "An unexpected error occured while processing the request. Tracking ID: '96a679dd-cdf8-4f8a-b1b3-5a54d6792569'"
    }
}

[pamelafox]

Okay so I just tried creating a PostGreSQL server from the command line and got an error:

>>> az group create --name postgres-test --location eastus2
>>> az postgres flexible-server create --resource-group postgres-test
Checking the existence of the resource group 'postgres-test'...
Resource group 'postgres-test' exists ? : True 
No available SKUs in this location

The pricing page indicates it's available in East US 2:

So there's some disconnect with the availability. It's also problematic that it gets reported as an InternalServerError versus the actual error, as this has taken me hours to narrow down.

[pamelafox]

I've pinged a PG Flex PM about this issue.

[pamelafox]

Okay, so apparently it is only a limitation for internal accounts, so external developers won't experience it. However, I would like to figure out at what point "No available SKUs in this location" turns into the undecipherable "Internal Server Error" to save on future hours debugging issues like that. That's likely not in this project, but if anyone knows an appropriate place for filing that, I'd appreciate a lead.

[rajeshkamal5050]

Okay, so apparently it is only a limitation for internal accounts, so external developers won't experience it. However, I would like to figure out at what point "No available SKUs in this location" turns into the undecipherable "Internal Server Error" to save on future hours debugging issues like that. That's likely not in this project, but if anyone knows an appropriate place for filing that, I'd appreciate a lead.

@jongio @ellismg any pointers on which team to get this re-directed and moved?

[jongio]

Please try with the latest build from main. We prop errors now. https://github.com/Azure/azure-dev/blob/main/cli/installer/README.md#download-from-daily-builds

[pamelafox]

Just tried!

It does return the whole JSON. Unfortunately, that JSON doesn't have the SKU error, it has the InternalServerError still. I don't think this is an issue for your tracker anymore, so feel free to close the issue, and I can keep looking for the right issue tracker.

Error: deploying infrastructure: error deploying infrastructure: failed deploying: deploying to subscription: GET https://management.azure.com/subscriptions/32ea8a26-5b40-4838-b6cb-be5c89a57c16/providers/Microsoft.Resources/deployments/django-sample16/operationStatuses/08585349506033408375
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
  "status": "Failed",
  "error": {
    "code": "DeploymentFailed",
    "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
    "details": [
      {
        "code": "Conflict",
        "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"Conflict\",\r\n            \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource operation completed with terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"InternalServerError\\\",\\r\\n        \\\"message\\\": \\\"An unexpected error occured while processing the request. Tracking ID: '150ee84f-938d-43da-bc70-6b1f894c7c23'\\\"\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
      }
    ]
  }
}
--------------------------------------------------------------------------------

[rajeshkamal5050]

@pamelafox thanks for the update. Please do cross-reference this issue when you get the other issue filed.

[pamelafox]

Haven't found an issue tracker for the underlying issue yet, but I did file an issue for this repo which could help with this particular eastus2 provisioning issue: https://github.com/Azure/azure-dev/issues/995

[jongio]

I'm going to re-open this as I just hit it as well. I think we need to follow up with the postgres team to see if we can get a better error message propagated from ARM.

[charris-msft]

Adding some recommendations here because I just hit the same issue - same as @pamelafox originally reported with InternalServerError and nothing useful.

My recommendations:

1. Provide a link to the portal where I can lookup the Tracking ID - for all such errors, regardless of RP.
2. Don't delete the progress step: |= | Creating/Updating resources (Azure Database for PostgreSQL flexible server) because it makes it harder for me to figure out which step caused the issue. Instead, mark it as (X) Failed: ... or something.
3. Get the PG team to surface a more useful error message (as you said)
4. Pre-validate the resource availability - I know PG said this wouldn't affect customers, but I find that hard to believe it won't ever impact them, and internal folks are people too - we don't want to try 11 times and spend hours troubleshooting.