`[Network/VPNGateway]` deployment operation failed due to an intermittent error

[wuxu92]

Brief

VPN gateway deployment keeps failing with helpless information

Description

When creating a virtual network gateway for VPN Type with route-based policy, the create operation returns with server-side error without debugging/helpful information to investigate. I have tried with different SKU types and modified the customeRoute prefix value, which makes no different and all failed.

related API:

https://github.com/Azure/azure-rest-api-specs/blob/ae227e20bcb22d83634452026f701d75bec2619e/specification/network/resource-manager/Microsoft.Network/stable/2021-08-01/virtualNetworkGateway.json#L1937

example

gateway request payload

{
    "location": "eastus",
    "properties": {
        "activeActive": false,
        "customRoutes": {
            "addressPrefixes": [
                "101.168.0.6/32"
            ]
        },
        "enableBgp": false,
        "enablePrivateIpAddress": true,
        "gatewayType": "Vpn",
        "ipConfigurations": [
            {
                "name": "vnetGatewayConfig",
                "properties": {
                    "privateIPAllocationMethod": "Dynamic",
                    "publicIPAddress": {
                        "id": "/subscriptions/xxx-xxxx/resourceGroups/xxx-rg/providers/Microsoft.Network/publicIPAddresses/vpnpubip001"
                    },
                    "subnet": {
                        "id": "/subscriptions/xxx-xxxx/resourceGroups/xxx-rg/providers/Microsoft.Network/virtualNetworks/vpngw001/subnets/GatewaySubnet"
                    }
                }
            }
        ],
        "sku": {
            "name": "VpnGw3AZ",
            "tier": "VpnGw3AZ"
        },
        "vpnType": "RouteBased"
    }
}

reponse error message:

{
    "error": {
        "code": "VmssGatewayDeploymentFailed",
        "details": [],
        "message": "The gateway deployment operation failed due to an intermittent error. Please try again."
    },
    "status": "Failed"
}

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @vpngwsuppgithub.

Issue Details

---

Brief

VPN gateway deployment keeps failing with helpless information

Description

When creating a virtual network gateway for VPN Type with route-based policy, the create operation returns with server-side error without debugging/helpful information to investigate. I have tried with different SKU types and modified the customeRoute prefix value, which makes no different and all failed.

related API:

https://github.com/Azure/azure-rest-api-specs/blob/ae227e20bcb22d83634452026f701d75bec2619e/specification/network/resource-manager/Microsoft.Network/stable/2021-08-01/virtualNetworkGateway.json#L1937

example

gateway request payload

{
    "location": "eastus",
    "properties": {
        "activeActive": false,
        "customRoutes": {
            "addressPrefixes": [
                "101.168.0.6/32"
            ]
        },
        "enableBgp": false,
        "enablePrivateIpAddress": true,
        "gatewayType": "Vpn",
        "ipConfigurations": [
            {
                "name": "vnetGatewayConfig",
                "properties": {
                    "privateIPAllocationMethod": "Dynamic",
                    "publicIPAddress": {
                        "id": "/subscriptions/xxx-xxxx/resourceGroups/xxx-rg/providers/Microsoft.Network/publicIPAddresses/vpnpubip001"
                    },
                    "subnet": {
                        "id": "/subscriptions/xxx-xxxx/resourceGroups/xxx-rg/providers/Microsoft.Network/virtualNetworks/vpngw001/subnets/GatewaySubnet"
                    }
                }
            }
        ],
        "sku": {
            "name": "VpnGw3AZ",
            "tier": "VpnGw3AZ"
        },
        "vpnType": "RouteBased"
    }
}

reponse error message:

{
    "error": {
        "code": "VmssGatewayDeploymentFailed",
        "details": [],
        "message": "The gateway deployment operation failed due to an intermittent error. Please try again."
    },
    "status": "Failed"
}

Author:	wuxu92
Assignees:	-
Labels:	Network - VPN Gateway, Service Attention, needs-triage
Milestone:	-

[mayank-reynencourt]

Hi,

i'm also trying to create VNG and face same issue , any update on this ?

[el-memer]

Hi, I'm also facing same issue here (Region West Europe), I've opened a support ticket and will send updates here if I have some.

[birdnathan]

Same here - West Europe. Please fix!

[FletchAD]

Same here - UKSouth and UKWest, tried with VpnGw1 & VpnGw1AZ, interestingly the basic SKU seems to work ok, but that's not what I need.

[slaffka-vlasov]

The same is for uswest3 with VpnGw1 & VpnGw2. Basic doesn't work as well

[FrankMormino]

Same thing here - for west Europe as well.

[FrankMormino]

Hi, I'm also facing same issue here (Region West Europe), I've opened a support ticket and will send updates here if I have some.

Same here - trying the same in West Europe - any luck on the support ticket side?

[ghost]

Same issue here. Happening in West Europe trying to deploy a VpnGw2AZ. Tried deployment via Terraform and the Portal.

[BevanSin]

Note there is an issue currently with Azure Key Vault that may be causing this problem - if you look in the Service Health page there is an alert there that is affecting downstream Azure services like VPN Gateway.

[birdnathan]

From Azure support: “Hi Nathan

Thank you for contacting Microsoft Azure Networking support.

Please know that we are currently investigating an ongoing global issue regarding Virtual Network Gateway (VPN Gateway) deployments and we will get back to you with updates once this has been mitigated.”

Seems the key vault issue is wiping out network services. The status website need to reflect this and not just list key vault as impacted

[haciz]

Hello,

Indeed joining the list to report an issue, neither the empty PUT request is helping

Set-AzVirtualNetworkGateway: Long running operation failed with status 'Failed'. Additional Info:'The gateway deployment operation failed due to an intermittent error. Please try again.' StatusCode: 200 ReasonPhrase: OK Status: Failed ErrorCode: VmssGatewayDeploymentFailed ErrorMessage: The gateway deployment operation failed due to an intermittent error. Please try again.

[imkevinjones]

Same issue here... This has been happening all day.

[shanyuen]

Yesterday I bought a subscription and created a fresh virtual network in southeast asia. But still error.

Status: Conflict {"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"VmssGatewayDeploymentFailed","message":"The gateway deployment operation failed due to an intermittent error. Please try again."}]} SKU: VpnGw1 Correlation id: 2f5880ba-a8b2-4184-801e-a5b8cea6f3b4

[FletchAD]

After this confirmation at 3rd Nov @ 05:52 UTC https://status.azure.com/en-us/status/history/ I was able to provision VpnGw1 in UKSouth this morning.

[ghost]

Seems to be mitigated. Can also deploy the VGW now.

[el-memer]

Got an answer from the support team.

I apologize for the inconvenience that was caused but as we checked, there is a service disruption in all regions, which impacts the deployments of the VPN Gateway resources. We hope that it might be resolved by the end of the day. Our internal team are actively investigating and working on a solution, however, currently we do not have a specific resolution date or time.

EDIT: My deployment succeeded this morning in Region West Europe for a VpnGw1 Route based !

I'll let you know if I have some updates from support, but indeed this seems to be mitigated.

[haciz]

Indeed it works on Express route gateways as well West Europe.

[slaffka-vlasov]

Confirmed creation of VpnGw1 in west us 3

[laglergruener]

Hi, we‘ve currently the same issue, mentioned above in WestEurope for AZ gateways. Is the problem still available? Thanks, Hannes

[rik-v]

This issue seems to be current again. We're currently unable to deploy any VPN gateway to any subscription or any tenant. Tried 3 different tenants, 4 different subscriptions, 2 types of SKU (VpnGw1 and VpnGw2), all in West Europe. We noticed this starting january 9, 2024, however, it is not unlikely this issue persisted before that date.

No mentions of this on the service health pages. So, curious as to what's causing it this time.

[soufianerabi]

Yeah, for the past three days, we've been facing the same issue in our organization. The deployment has failed multiple times, even after many retries. Unfortunately, Microsoft support wasn't helpful. We're still trying to deploy this resource, but it continues to fail.

[tanarchytan]

Also a problem on my side. tried it since 09 of January. Region West Europe with SKU VPNGw2

{ "code": "DeploymentFailed", "target": "/subscriptions/[...]/resourceGroups/rg-net-hub/providers/Microsoft.Resources/deployments/Microsoft.Template-20240115105923", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "ResourceDeploymentFailure", "target": "/subscriptions/[...]/resourceGroups/rg-net-hub/providers/Microsoft.Network/virtualNetworkGateways/vpng-[...]-prod", "message": "The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'." } ] }

[rik-v]

@soufianerabi @tanarchytan, I tried deploying in north europe last friday (jan. 12th, 2024), which completes successfully, which makes me believe Microsoft simply has some serious resource issues in the west europe region.

It's not the first time (or sign of) this is happening, a few months back we were unable to roll out other resources (VM's, VMSS'es (for AKS)) because apparently the resource was unavailable in a specific zone in west europe. We ended up deploying to "just" 2 zones instead of 3 (still ok, but we wanted 3 :)). To be clear: we haven't checked if those resources have become available again in all zones since then, so perhaps things improved (although I strongly doubt that considering the fact we can't deploy VPN to west europe)

Would be helpful if Microsoft would simply clarify the current state of west europe, if there's a capacity issue, that's annoying, but I'd rather have them tell us then having to find out this way...

[TGosselink]

Some here, VPN deploy in West Europe: The gateway deployment operation failed due to an intermittent error. Please try again. (Code: VmssGatewayDeploymentFailed)

[lgriffithsdoherty]

Me too

[foxmeyson]

Microsoft is not good, there are problems all the time. That's horrible

[rbnmk]

Currently facing same issue in West Europe for VpnGw1 and VpnGw1AZ

[mtc3net]

Me too

[lgriffithsdoherty]

Microsoft have called me to say that they are not sure what the issue is and it could be months before it is resolved. The attached states capacity constraints but still no resolution in site just a warning set up to say capacity constraints wont allow you to build gateways in west europe for the foreseeable.