Retry "AnotherOperationInProgress" error messages

[sam-cogan]

Affected feature

Currently if Azure generates a "AnotherOperationInProgress" error Pulumi will fail. This error is often caused by trying to access two sub-resources on a resource at the same time. A good example is creating multiple subnets on a virtual network. Pulumi will try and create them all that the same time, but Azure can only create them one at a time.

Adding a back-off and retry for the "AnotherOperationInProgress" error message would help workaround this problem

[infin8x]

@sam-cogan we'll take a look at this. In the meantime, if you create a dependsOn chain with these sub-resources, Pulumi won't try to create them all at the same time. See https://www.pulumi.com/docs/intro/concepts/resources/#dependson for details.

[sam-cogan]

@infin8x thanks, I have done that and it does indeed work around the issue.

[masashi-shib]

@sam-cogan we'll take a look at this. In the meantime, if you create a dependsOn chain with these sub-resources, Pulumi won't try to create them all at the same time. See https://www.pulumi.com/docs/intro/concepts/resources/#dependson for details.

While this workaround works perfectly the chaining of dependencies forces the developer to create the whole VNET and its Subnets in one module which is not optimal especially when trying to split code into different modules.

One example might be that the code for a specific module should create the Subnet in the VNET add the corresponding resources and configure firewall rules.

[masashi-shib]

@mikhailshilkov @infin8x I would also love to hear what is the impact of deleting the first subnet only (e.g. replacing its name) after the chaining is done.

e.g. Subnet 1 <--depends_on-- Subnet2 <--depends_on-- Subnet 3

What happens if in the case the Subnet 1 name has been changed? Does it create a havox delete basically on all resources that have an assignment in any of the subnets?

[mikhailshilkov]

Does it create a havox delete basically on all resources that have an assignment in any of the subnets?

I think the answer is yes, unfortunately. In that case you'd have to remove all the dependsOn annotations and run an update before trying to rename the first subnet. I realise it's not ideal and we should solve the original problem.

[deepak-goel]

Hi team, are there any plans to fix this? Terraform also had the same kind of race condition, but they already fixed that a while back.

[MatthewBettonMaif]

Hello Team, We are having exactly the same problem and we have to remove our factorization in order to chain the creation of the subnets in same vNet by the code :/

[DSTAlan]

I'm also running into this issue so a fix would be great, Thanks

Alan

[RockyMM]

We just noticed this problem. If you need any help with this, I am more that willing!

[olafurnielsen]

Any updates on this issue? It's really unfortunate to have this awkward limitation, especially when Pulumi is being deployed via CI/CD and I don't feel like chaning depends is a viable option.

[RDavis3000]

I'm getting it when trying to deploy a webapp. It deployed the rest of the stack ok, but keeps failing on this webapp, even though its the ONLY thing being deployed! I've waited 10-15 minutes from the first attempt.

its my first time using Pulumi so possibly I am doing something wrong:

// Create FunctionApp with Application Insights 
var functionApp = new WebApp(functionAppName, new WebAppArgs
{
    ResourceGroupName = resourceGroup.Name,
    ServerFarmId = appServicePlan.Id,
    SiteConfig = new SiteConfigArgs
    {
        AppSettings = new[]
        {
                new NameValuePairArgs { Name = "APPINSIGHTS_INSTRUMENTATIONKEY", Value = appInsights.InstrumentationKey },
                new NameValuePairArgs { Name = "WEBSITE_RUN_FROM_PACKAGE", Value = @"1" },
                new NameValuePairArgs { Name = "FUNCTIONS_WORKER_RUNTIME", Value = "dotnet" },
        },
        AlwaysOn = true,
    },
    Identity = new ManagedServiceIdentityArgs { Type = ManagedServiceIdentityType.SystemAssigned }
}, new CustomResourceOptions()
{
    DependsOn = new InputList<Pulumi.Resource>() { appInsights, imageStorageAccount, storageAccount, appServicePlan, blobContainer }
});

getting:

 Type                        Name         Status                  Info
 pulumi:pulumi:Stack         BandLab-dev  **failed**              1 error

• └─ azure-native:web:WebApp functionApp creating failed 1 error Diagnostics: pulumi:pulumi:Stack (BandLab-dev): error: update failed

azure-native:web:WebApp (functionApp): error: autorest/azure: Service returned an error. Status= . Cannot modify this site because another operation is in progress. Details: Id: 763f49a6-6c9d-474b-8308-5b47e4df80fb, OperationName: Create, CreatedTime: 5/14/2024 9:08:39 PM, RequestId: 71a73020-cef3-475a-89ed-c7af7e177931, EntityType: 3

[j-fulbright]

I'm getting it when trying to deploy a webapp. It deployed the rest of the stack ok, but keeps failing on this webapp, even though its the ONLY thing being deployed! I've waited 10-15 minutes from the first attempt.

its my first time using Pulumi so possibly I am doing something wrong:

// Create FunctionApp with Application Insights 
var functionApp = new WebApp(functionAppName, new WebAppArgs
{
    ResourceGroupName = resourceGroup.Name,
    ServerFarmId = appServicePlan.Id,
    SiteConfig = new SiteConfigArgs
    {
        AppSettings = new[]
        {
                new NameValuePairArgs { Name = "APPINSIGHTS_INSTRUMENTATIONKEY", Value = appInsights.InstrumentationKey },
                new NameValuePairArgs { Name = "WEBSITE_RUN_FROM_PACKAGE", Value = @"1" },
                new NameValuePairArgs { Name = "FUNCTIONS_WORKER_RUNTIME", Value = "dotnet" },
        },
        AlwaysOn = true,
    },
    Identity = new ManagedServiceIdentityArgs { Type = ManagedServiceIdentityType.SystemAssigned }
}, new CustomResourceOptions()
{
    DependsOn = new InputList<Pulumi.Resource>() { appInsights, imageStorageAccount, storageAccount, appServicePlan, blobContainer }
});

getting:

 Type                        Name         Status                  Info
 pulumi:pulumi:Stack         BandLab-dev  **failed**              1 error

• └─ azure-native:web:WebApp functionApp creating failed 1 error Diagnostics: pulumi:pulumi:Stack (BandLab-dev): error: update failed

azure-native:web:WebApp (functionApp): error: autorest/azure: Service returned an error. Status= . Cannot modify this site because another operation is in progress. Details: Id: 763f49a6-6c9d-474b-8308-5b47e4df80fb, OperationName: Create, CreatedTime: 5/14/2024 9:08:39 PM, RequestId: 71a73020-cef3-475a-89ed-c7af7e177931, EntityType: 3

Is your SitePlan for this function app a Consumption plan? If so, alwaysOn wil break it.

[smith558]

Still an issue.