Service Bus trigger using System Assigned MI not triggering due to FunctionListenerException

[willvelida]

trafficstars

Investigative information

Please provide the following:

• Timestamp: 2:11:52 PM (NZT)
• Function App version: 4.8.0.18714
• Function App name: sbmi28972fa
• Function name(s) (as appropriate): ProcessOrderFromQueue
• Region: Australia East

I have a ServiceBus Trigger function written in C# that's failing to start due to the following exception:

Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException: The listener for function 'ProcessOrderFromQueue' was unable to start.
 ---> System.FormatException: The connection string could not be parsed; either it was malformed or contains no well-known tokens.
   at Azure.Messaging.ServiceBus.ServiceBusConnectionStringProperties.Parse(String connectionString)
   at Azure.Messaging.ServiceBus.ServiceBusConnection..ctor(String connectionString, ServiceBusClientOptions options)
   at Azure.Messaging.ServiceBus.ServiceBusClient..ctor(String connectionString, ServiceBusClientOptions options)
   at Microsoft.Azure.WebJobs.ServiceBus.MessagingProvider.<>c__DisplayClass7_0.<CreateClient>b__0(String _)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Microsoft.Azure.WebJobs.ServiceBus.MessagingProvider.CreateClient(String connectionString, ServiceBusClientOptions options)
   at Microsoft.Azure.WebJobs.Extensions.ServiceBus.Config.ServiceBusClientFactory.CreateClientFromSetting(String connection)
   at Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener.<>c__DisplayClass21_0.<.ctor>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener.<>c__DisplayClass21_0.<.ctor>b__2()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(CancellationToken cancellationToken, Boolean allowRetry) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs:line 68
   --- End of inner exception stack trace ---

To interact with Service Bus, I'm using Managed Identities and a combination of instantiating a ServiceBusClient in my Startup.cs file, as well as using a Service Bus Trigger to read messages off a queue. I have set up the following configuration for my Function App (infrastructure has been provisioned using Terraform):

app_settings = {
      "APPINSIGHTS_INSTRUMENTATIONKEY" = "${azurerm_application_insights.appins.instrumentation_key}"
      "APPLICATIONINSIGHTS_CONNECTION_STRING" = "InstrumentationKey=${azurerm_application_insights.appins.instrumentation_key};IngestionEndpoint=https://australiaeast-1.in.applicationinsights.azure.com/;LiveEndpoint=https://australiaeast.livediagnostics.monitor.azure.com/"
      "QueueName" = "${azurerm_servicebus_queue.ordersqueue.name}"
      "ServiceBusConnection__fullyQualifiedNamespace" = "${azurerm_servicebus_namespace.sbnamespace.name}.servicebus.windows.net"
      "ServiceBusConnection__credential" = "managedIdentity"
      "ServiceBusConnection" = "${azurerm_servicebus_namespace.sbnamespace.name}.servicebus.windows.net"
    }

The following role assignments have been applied in my Terraform script:

resource "azurerm_role_assignment" "sbrole" {
    scope = azurerm_servicebus_namespace.sbnamespace.id
    role_definition_name = "Azure Service Bus Data Owner"
    principal_id = azurerm_windows_function_app.funcapp.identity.0.principal_id
}

resource "azurerm_role_assignment" "sbreaderrole" {
  scope = azurerm_servicebus_namespace.sbnamespace.id
    role_definition_name = "Azure Service Bus Data Receiver"
    principal_id = azurerm_windows_function_app.funcapp.identity.0.principal_id
}

I am able to send messages to the queue successfully (using the ServiceBusClient in my Startup.cs file). However, my Function using a ServiceBus Trigger doesnt start due to the above exception. So my message stays in the queue.

I have followed the guidance here on setting us my Function app to use MSI to authenticate with Service Bus, so I'm wondering if there is a bug in the Trigger that's causing it to not start, or whether I've missed something in the setup?

Repro steps

Provide the steps required to reproduce the problem:

1. Clone the following repository
2. Provision the terraform file (terraform apply)
3. Publish the function code to the Function App
4. Send a POST request to the SendOrderToQueue function.
5. Observe exception being thrown in App Insights Live Metrics.

Expected behavior

The function should be able to log the content of the queue defined in the function ProcessOrderFromQueue.

Actual behavior

Exception above is thrown is Application Insights Live Metrics

Known workarounds

Use Connection String of Service Bus namespace, instead of using MSI.

Related information

Provide any related information

• Programming language used = C#
• Links to source = https://github.com/willvelida/azure-functions-service-bus-msi
• Bindings used = Service Bus (version 5.6.0)

[willvelida]

I've managed to fix the underlying issue. I changed the App Setting that the ServiceBusClient uses in the Startup.cs file to ServiceBusEndpoint and the Service Bus Trigger now successfully starts.

I'm keeping this open since I'm not sure why this works. How is the SDK interpreting the hostname when connecting to the client that would cause this error to surface? It seems a strange that we'd need two app settings with the same hostname value to get this to work.

[erionpc]

It seems that there is a "magic string" in play here. The setting name which defines the service bus namespace:

1. MUST NOT be a nested setting (i.e. must not contain any ":" characters)
2. MUST END with "__fullyQualifiedNamespace"

The setting name in the ServiceBusTrigger definition, however, must not include the magic string __fullyQualifiedNamespace.

So:

code

FunctionName("MyFunction")]
public async Task Run([ServiceBusTrigger("%MyQueueSettingName%", Connection = "MyConnectionSettingName")]

setting

"MyConnectionSettingName__fullyqualifiednamespace": "mybus.servicebus.windows.net"

This is pretty nasty :)

[audunsolemdal]

Can confirm that the above method described by @erionpc works. Glad I found this early or it would have had the potential for my head bashing the wall. Note that new ServicebusClient() works just fine without this black magic.