Azure Function slot killing Orphan processes after swap

[RaphaelEckmayr]

Problem

I am currently working on a prototype for using Azure Function on App Service Plan with a Service Bus trigger and a staging slot. The issue is that after the swap, all running processes are failing. In our production environment we have Functions that can run up to 15 minutes and CANNOT have a retry logic and also contain Durable Functions. For me it looks like, after the swap the function fails to lock the message again. The Question is, is this behavior expected? If not, what would I need to change to fix it?

Code:

Everything in <> is just a placeholder and is filled in on the real thing.

Bicep:

var location = resourceGroup().location
var appInsightsLocation =  resourceGroup().location
var storageAccountType = 'Standard_LRS'
var functionAppName = '<Function Name>'
var hostingPlanName = '<App Service Plan Name>'
var busName = functionAppName
var applicationInsightsName = '<App Insights Name>'
var storageAccountName = '<Storage Account Name>'
var topicName = '${functionAppName }-topic'
var subscriptionName = '${functionAppName }-subs'
var shared_func_rg = resourceGroup(subscription().subscriptionId, '<Resource Group name of App Service Plan>')

var service_func_package = '<Link to ZIP of the Function>'



resource storageAccount 'Microsoft.Storage/storageAccounts@2022-05-01' = {
  name: storageAccountName
  location: location
  sku: {
    name: storageAccountType
  }
  kind: 'Storage'
  properties: {
    supportsHttpsTrafficOnly: true
    defaultToOAuthAuthentication: true
  }
}

resource hostingPlan 'Microsoft.Web/serverfarms@2020-06-01' existing = {
  name: hostingPlanName
  scope: shared_func_rg
}

resource function 'Microsoft.Web/sites@2022-09-01' = {
  name: functionAppName
  location: resourceGroup().location
  kind: 'functionapp'
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    alwaysOn: true
    serverFarmId: hostingPlan.id
  }
}

module appSettings 'appendAppSettings.bicep' = {
  name: 'appSettings'
  params: {
    currentAppSettings: list('${function.id}/config/appsettings', '2022-09-01').properties
    appSettings: {
      'FUNCTIONS_EXTENSION_VERSION': '~4'
      'WEBSITE_ADD_SITENAME_BINDINGS_IN_APPHOST_CONFIG': '1'
    }
    appName: functionAppName
  }
}

resource slot 'Microsoft.Web/sites/slots@2016-08-01' = {
  name: 'slot'
  location: resourceGroup().location
  kind: 'functionapp'
  parent: function
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    httpsOnly: true
    enabled: true
    alwaysOn: true
    serverFarmId: hostingPlan.id
    siteConfig: {
      appSettings: [
        {
          name: 'WEBSITE_ADD_SITENAME_BINDINGS_IN_APPHOST_CONFIG'
          value: 1
        }
        //Access to ServiceBus via Managed Identity
        {
          name: 'WEBSITE_RUN_FROM_PACKAGE'
          value: service_func_package
        }
        {
          name: 'ServiceBusConnection__fullyQualifiedNamespace'
          value: '<Connection String>'
        }
        {
          name: 'AzureWebJobsStorage'
          value: 'DefaultEndpointsProtocol=https;AccountName=${storageAccountName};AccountKey=${storageAccount.listKeys().keys[0].value}'
        }
        {
          name: 'ApplicationDataStorage'
          value: 'DefaultEndpointsProtocol=https;AccountName=${storageAccountName};AccountKey=${storageAccount.listKeys().keys[0].value}'
        }
        {
          name: 'FUNCTIONS_EXTENSION_VERSION'
          value: '~4'
        }
        {
          name: 'APPINSIGHTS_INSTRUMENTATIONKEY'
          value: applicationInsights.properties.InstrumentationKey
        }
        {
          name: 'FUNCTIONS_WORKER_RUNTIME'
          value: 'dotnet'
        }
        {
          name: 'WEBSITE_OVERRIDE_PRESERVE_DEFAULT_STICKY_SLOT_SETTINGS'
          value: 'false'
        }
      ]
      minTlsVersion: '1.2'
    }
  }
}

resource applicationInsights 'Microsoft.Insights/components@2020-02-02' = {
  name: applicationInsightsName
  location: appInsightsLocation
  kind: 'web'
  properties: {
    Application_Type: 'web'
    Request_Source: 'rest'
  }
}

resource bus 'Microsoft.ServiceBus/namespaces@2018-01-01-preview' = {
  name: busName
  location: location
  sku: {
    name: 'Standard'
  }
  properties: {
    zoneRedundant: false
  }
}

resource topic 'Microsoft.ServiceBus/namespaces/topics@2018-01-01-preview' = {
  parent: bus
  name: topicName
  properties: {}
}


resource sub_function'Microsoft.ServiceBus/namespaces/topics/subscriptions@2018-01-01-preview' = {
  parent: topic
  name: subscriptionName
  properties: {
    maxDeliveryCount: 1
  }
}

resource owner_role_assignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(function.id, bus.id, topic.id, '<RoleId>')
  scope: topic
  properties: {
    roleDefinitionId: tenantResourceId('Microsoft.Authorization/roleDefinitions', '<RoleId>')
    principalId: function.identity.principalId
  }
}

resource sender_role_assignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(function.id, bus.id, topic.id, '<RoleId>')
  scope: topic
  properties: {
    roleDefinitionId: tenantResourceId('Microsoft.Authorization/roleDefinitions', '<RoleId>')
    principalId: function.identity.principalId
  }
}

Module:

param currentAppSettings object 
param appSettings object
param appName string

resource siteconfig 'Microsoft.Web/sites/config@2020-12-01' = {
  name: '${appName}/appSettings'
  properties: union(currentAppSettings, appSettings)
}

Function:

using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;

namespace ServiceBusFunctions
{
    public class ServiceBusReceiver
    {
        private readonly ILogger<ServiceBusReceiver> _logger;

        public ServiceBusReceiver(ILogger<ServiceBusReceiver> log)
        {
            _logger = log;
        }

        [FunctionName("ServiceBusReceiver")]
        public async Task Run([ServiceBusTrigger("<Topic Name>", "<Subscription Name>", Connection = "ServiceBusConnection")] string mySbMsg)
        {
            //Delay 5 min
            await Task.Delay(300000);
            _logger.LogInformation("Received");
        }
    }
}

Pipeline:

The pipeline are just a few simple powershell scripts which start and swap and stop the slot.

Other things I have tried:

• Giving the Slot access to the Service Bus
• Add and remove 'WEBSITE_ADD_SITENAME_BINDINGS_IN_APPHOST_CONFIG': '1'
• Thread.Sleep in the Function (in async and sync functions)
• Removing the appendAppsettings module
• Using a Shared Access Key to access the Servie Bus

Logs:

[bhagyshricompany]

Thanks for repoting please share the func app name,invocation id,timestamp etc.

[bhagyshricompany]

NO Updated hence closing