Aspire Storage integration does not work due to ;; part in connection string

[pksorensen]

Is there an existing issue for this?

• [x] I have searched the existing issues

Describe the bug

Looking at: https://github.com/dotnet/aspire/blob/main/src/Aspire.Hosting.Azure.Storage/AzureStorageEmulatorConnectionString.cs

the endpoint for emulator ends with ;

builder.Append($"{key}={endpoint.Property(EndpointProperty.Scheme)}://{endpoint.Property(EndpointProperty.IPV4Host)}:{endpoint.Property(EndpointProperty.Port)}/devstoreaccount1;");

and then the queue connection string gets ;; due to https://github.com/dotnet/aspire/blob/main/src/Aspire.Hosting.Azure.Storage/AzureQueueStorageResource.cs

https://github.com/dotnet/aspire/blob/main/src/Aspire.Hosting.Azure.Storage/AzureQueueStorageResource.cs

which then ends up with a connection string xxx;;queuename and the azure queue service client is not able to parse that.

Expected Behavior

The expected behavior is that when adding the azure service client to an application that reference the queue, it can connect instead of crashing.

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version info

No response

Anything else?

No response

[davidfowl]

What does your client code look like?

[pksorensen]

AppHost

// Add Azure Storage emulator with fixed port assignments for WSL compatibility
var azureStorage = builder
    .AddAzureStorage("blob-storage")
    .RunAsEmulator(resourceBuilder =>
    {
        resourceBuilder.WithDataVolume($"{prefix}-azure-storage-data");
        resourceBuilder.WithBlobPort(10000);
        resourceBuilder.WithLifetime(ContainerLifetime.Persistent);
    }
    );

var blobs = azureStorage
    .AddBlobs("blobs");

azureStorage
    .AddBlobContainer("profiles");
azureStorage
    .AddBlobContainer("data");

var queue = azureStorage.AddQueue("blob-events");

and service

builder.AddAzureQueueServiceClient("blob-events");

// Register Azure Storage implementations
builder.Services.AddSingleton<MaaS.BusinessLogic.Queuing.IStorageQueue>(sp =>
{
    var queueServiceClient = sp.GetRequiredService<Azure.Storage.Queues.QueueServiceClient>();
    var queueClient = queueServiceClient.GetQueueClient("blob-events");
    return new MaaS.BusinessLogic.Queuing.AzureStorageQueue(queueClient);
});

[MihuBot]

I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):

• https://github.com/dotnet/aspire/issues/10671

[davidfowl]

What's the exception you're seeing?

[pksorensen]

The queue service clients cant parse DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;QueueEndpoint=http://127.0.0.1:52537/devstoreaccount1;;QueueName=blob-events

Unhandled exception. System.FormatException: No valid combination of account information found.
    at Azure.Storage.StorageConnectionString.<>c.<Parse>b__67_0(String err)
    at Azure.Storage.StorageConnectionString.ParseCore(String connectionString, StorageConnectionString& accountInformation, Action`1 error)
    at Azure.Storage.StorageConnectionString.Parse(String connectionString)
    at Azure.Storage.Queues.QueueServiceClient..ctor(String connectionString, QueueClientOptions options)

[pksorensen]

So if there is a workaround to strip the ``;;` part until fixed that would be much appreciated :D Assuming i am right that its a bug, but also wondering why no one else have hit this, so wonder what i can be doing wrong

[davidfowl]

but also wondering why no one else have hit this, so wonder what i can be doing wrong

Yes, this part is surprising 😄.

cc @sebastienros @eerhardt

[pksorensen]

Would you have a hack/workaround to manipuate the connection string in apphost?

[pksorensen]

I made a hack

    .WithEnvironment("ConnectionStrings__blob-events", () =>
    {
        // This callback will be executed when the connection string is actually resolved
        var task = queue.Resource.ConnectionStringExpression.GetValueAsync(CancellationToken.None);
        var connectionString = task.GetAwaiter().GetResult();
        return connectionString.Replace(";;", ";");
    })

giving this connection string

DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;QueueEndpoint=http://127.0.0.1:55698/devstoreaccount1;QueueName=blob-events

but that just gave same result,

so not a fix to fix ;;

Need to do some tests with queue service client and connection strings then.

[pksorensen]

private async Task<string?> GetConnectionStringAsync(IResource resource, CancellationToken cancellationToken)
{
    // For Azure Storage resources, get the connection string
    if (resource is IResourceWithConnectionString connectionStringResource)
    {
        var conn = await connectionStringResource.GetConnectionStringAsync(cancellationToken);

        if(!string.IsNullOrEmpty(conn))
        {
            // For queue resources, Aspire appends ";QueueName=xxx" to the connection string
            // We need to remove this as it's not valid for QueueServiceClient (only for QueueClient)
            var queueNameIndex = conn.IndexOf(";QueueName=", StringComparison.OrdinalIgnoreCase);
            if (queueNameIndex >= 0)
            {
                conn = conn.Substring(0, queueNameIndex);
            }

            // Remove any trailing semicolons
            conn = conn.TrimEnd(';');

            return conn;
        }

    }

    await Task.CompletedTask;
    return null;
}

Above works - still need to figure out if we can do specific connection strings for specific queues for queue service client, i can understand why not and thats properly also something that should be fixed in aspire. The motivation behind appending QueueName= to connection string is interesting here in aspire code.

[pksorensen]

Aha moment

var queues = azureStorage
    .AddQueues("queues");

produces a valid connection string and then at the application level it can be pointed at the actual queue

[essenbee2]

Saved my sanity! Thanks for the solution @pksorensen 💯

[eerhardt]

The problem here is there are 2 similar resources:

• Storage Queues Service (the parent of all the individual queues)
• Storage Queue (an individual queue)

When you reference one of these resources, you need to match up the types between the apphost and the app. You either want:

Queues Service

AppHost

var queues = azureStorage.AddQueues("queues");

App

builder.AddAzureQueueServiceClient("queues");

-- or --

Individual Queue

AppHost

var queue = azureStorage.AddQueue("blob-events");

App

builder.AddAzureQueue("blob-events");

---

Mix and matching will cause issues because the App client expects the connection string to be correct for its type.

[ian-gavurin]

Thanks @eerhardt, that is indeed the issue. I spent ages on this today. I was passing in an individual queue.

As I wanted a QueueClient I had initially tried to use builder.AddAzureQueueClient, which seemed the obvious choice, but it's obsoleted, and advises use of builder.AddAzureQueueServiceClient instead, which injects QueueServiceClient.

builder.AddAzureQueue is needed to create a QueueClient, which I hadn't noticed, as it's running outside the naming convention, presumably because the old method creates a conflict.

Improvement options:

1. Given the clear intent to avoid the historical conflict, it's not easy to fix, there's a naming issue here causing a confusion. Maybe the error handler could be improved to check for this possibility, and provided clear feedback.

2. This isn't helped by the docs, which still direct you to the obsoleted method: https://learn.microsoft.com/en-us/dotnet/aspire/storage/azure-storage-queues-integration?tabs=dotnet-cli#add-azure-queue-storage-client - it could be clarified there too.