dotnet
Using Aspire.Microsoft.Extensions.Configuration.AzureAppConfiguration produces runtime error when deploying to Azure Container Apps
Is there an existing issue for this?
- [x] I have searched the existing issues
Describe the bug
Context
In our .NET Aspire project we just started using the Aspire.Microsoft.Extensions.Configuration.AzureAppConfiguration library version: 9.4.0-preview.1.25378.8.
This Azure App Configuration is an existing resource in our infrastructure and should be referenced in our API.
Currently our .NET Aspire AppHost project contains an API and an Azure Function project. These run fine locally, no issues whatsoever. All values including keyvault references are retrieved correctly. However, when deploying to Azure Container Apps, the container (API project) where the AppConfiguration is used, fails to start with an error message mentioned in the Exceptions section.
It would be fine if this problem can be looked into and addressed.
Please let me know if there are any missing things / parts in this issue.
Regards,
Expected Behavior
Integration with Azure App Configuration in .NET Aspire retrieves the configured configuration from that AppConfiguration and can be used in the application.
Steps To Reproduce
As I was wondering if the problem is related to any of our other software in the application, I just checkout the following repository and integrate the Aspire.Microsoft.Extensions.Configuration.AzureAppConfiguration package with 9.4.0-preview.1.25378.8 in it. It doesn't matter if we are using Aspire 9.3.1 or 9.4.0 as it produces the same error when deploying. Locally it runs just fine.
Exceptions (if any)
The exception we get when the container trying to start:
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'System.Memory.Data, Version=8.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
stderr F
stderr F File name: 'System.Memory.Data, Version=8.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'
stderr F at Azure.Data.AppConfiguration.ConfigurationClient.ParseGetConfigurationSettingsResponse(Response response)
stderr F at Azure.Data.AppConfiguration.ConditionalPageableImplementation.TryGetItemsFromResponse(Response response, String& nextLink, List`1& items)
stderr F at Azure.Data.AppConfiguration.ConditionalPageableImplementation.CreatePage(Response response, String& nextLink)
stderr F at Azure.Data.AppConfiguration.ConditionalPageableImplementation.AsPagesAsync(IEnumerable`1 conditionsEnumerable, String continuationToken, Nullable`1 pageSizeHint, CancellationToken cancellationToken)+MoveNext()
stderr F at Azure.Data.AppConfiguration.ConditionalPageableImplementation.AsPagesAsync(IEnumerable`1 conditionsEnumerable, String continuationToken, Nullable`1 pageSizeHint, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
stderr F at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass39_2.<<LoadSelected>b__0>d.MoveNext()
stderr F --- End of stack trace from previous location ---
stderr F at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass39_2.<<LoadSelected>b__0>d.MoveNext()
stderr F --- End of stack trace from previous location ---
stderr F at Microsoft.Extensions.Configuration.AzureAppConfiguration.TracingUtils.CallWithRequestTracing(Boolean tracingEnabled, RequestType requestType, RequestTracingOptions requestTracingOptions, Func`1 clientCall)
stderr F at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.LoadSelected(ConfigurationClient client, Dictionary`2 kvEtags, Dictionary`2 ffEtags, IEnumerable`1 selectors, HashSet`1 ffKeys, CancellationToken cancellationToken)
stderr F at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass38_0.<<InitializeAsync>b__0>d.MoveNext()
stderr F --- End of stack trace from previous location ---
stderr F at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass50_0.<<ExecuteWithFailOverPolicyAsync>b__0>d.MoveNext()
stderr F --- End of stack trace from previous location ---
stderr F at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.ExecuteWithFailOverPolicyAsync[T](IEnumerable`1 clients, Func`2 funcToExecute, CancellationToken cancellationToken)
stderr F at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.ExecuteWithFailOverPolicyAsync(IEnumerable`1 clients, Func`2 funcToExecute, CancellationToken cancellationToken)
stderr F at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.InitializeAsync(IEnumerable`1 clients, CancellationToken cancellationToken)
stderr F at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.TryInitializeAsync(IEnumerable`1 clients, List`1 startupExceptions, CancellationToken cancellationToken)
stderr F at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.LoadAsync(Boolean ignoreFailures, CancellationToken cancellationToken)
stderr F at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.Load()
stderr F at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)
stderr F at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)
stderr F at Microsoft.Extensions.Configuration.AzureAppConfigurationExtensions.AddAzureAppConfiguration(IConfigurationBuilder configurationBuilder, Action`1 action, Boolean optional)
stderr F at Microsoft.Extensions.Hosting.AspireAppConfigurationExtensions.AddAzureAppConfiguration(IHostApplicationBuilder builder, String connectionName, Action`1 configureSettings, Action`1 configureOptions) in /_/src/Components/Aspire.Microsoft.Extensions.Configuration.AzureAppConfiguration/AspireAppConfigurationExtensions.cs:line 56
.NET Version info
Locally:
.NET SDK:
Version: 9.0.302
Commit: bb2550b9af
Workload version: 9.0.300-manifests.2b366b05
MSBuild version: 17.14.13+65391c53b
Runtime Environment:
OS Name: Windows
OS Version: 10.0.26100
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\9.0.302\
.NET workloads installed:
[ios]
Installation Source: VS 17.14.36202.13
Manifest Version: 18.4.9288/9.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.ios\18.4.9288\WorkloadManifest.json
Install Type: Msi
[maui-windows]
Installation Source: VS 17.14.36202.13
Manifest Version: 9.0.51/9.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.maui\9.0.51\WorkloadManifest.json
Install Type: Msi
[android]
Installation Source: VS 17.14.36202.13
Manifest Version: 35.0.61/9.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.android\35.0.61\WorkloadManifest.json
Install Type: Msi
[maccatalyst]
Installation Source: VS 17.14.36202.13
Manifest Version: 18.4.9288/9.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.maccatalyst\18.4.9288\WorkloadManifest.json
Install Type: Msi
[aspire]
Installation Source: VS 17.14.36202.13
Manifest Version: 8.2.2/8.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.sdk.aspire\8.2.2\WorkloadManifest.json
Install Type: Msi
Configured to use loose manifests when installing new manifests.
Host:
Version: 9.0.7
Architecture: x64
Commit: 3c298d9f00
.NET SDKs installed:
8.0.311 [C:\Program Files\dotnet\sdk]
9.0.300 [C:\Program Files\dotnet\sdk]
9.0.302 [C:\Program Files\dotnet\sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.18 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]
Environment variables:
Not set
global.json file:
D:\playground\AzureAZDCli\aspire-samples\global.json
Learn more:
https://aka.ms/dotnet/info
Download .NET:
https://aka.ms/dotnet/download
Anything else?
AppHost.csproj:
<Project Sdk="Microsoft.NET.Sdk">
<Sdk Name="Aspire.AppHost.Sdk" Version="9.4.0" />
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net9.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<IsAspireHost>true</IsAspireHost>
<UserSecretsId>6f1ab4e9-b860-4044-9acd-ba7628231167</UserSecretsId>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Aspire.Hosting.AppHost" Version="9.4.0" />
<PackageReference Include="Aspire.Hosting.Azure.AppConfiguration" Version="9.4.0" />
<PackageReference Include="Aspire.Hosting.Azure.AppContainers" Version="9.4.0" />
<PackageReference Include="Aspire.Hosting.Azure.ApplicationInsights" Version="9.4.0" />
<PackageReference Include="Aspire.Hosting.Azure.Functions" Version="9.4.0-preview.1.25378.8" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\API\API1.csproj" />
<ProjectReference Include="..\Functions\Function1.csproj" />
</ItemGroup>
</Project>
API.csproj:
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net9.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<UserSecretsId>b2d60e6e-f213-4205-a343-5ec0d439ee56</UserSecretsId>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Flurl.Http" Version="4.0.2" />
<PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="9.0.7" />
<PackageReference Include="GeoJSON.Text" Version="1.1.0" />
<PackageReference Include="Aspire.Microsoft.Extensions.Configuration.AzureAppConfiguration" Version="9.4.0-preview.1.25378.8" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\AppHost.ServiceDefaults\AppHost.ServiceDefaults.csproj" />
</ItemGroup>
</Project>
AppHost's Program.cs (snippet)
IResourceBuilder<AzureApplicationInsightsResource> applicationInsights = builder.AddAzureApplicationInsights("application-insights");
IResourceBuilder<IResourceWithConnectionString> appConfiguration = builder.AddConnectionString("appconfig");
IResourceBuilder<ParameterResource> parameterInstanceId = builder.AddParameter("InstanceId");
IResourceBuilder<ParameterResource> geodataServerBaseUrl = builder.AddParameter("GeodataServerBaseUrl");
builder.AddAzureContainerAppEnvironment("cae-parkinglayertool")
.WithAzdResourceNaming();
IResourceBuilder<ProjectResource> container = builder.AddProject<Projects.API>("parking-layer-app")
.WithEnvironment("INSTANCE_ID", parameterInstanceId)
.WithEnvironment("GEODATA_SERVER_BASE_URL", geodataServerBaseUrl)
.WithReference(appConfiguration)
.WithReference(applicationInsights);
AddAzureAppConfiguration definition in the API Program.cs:
builder.AddAzureAppConfiguration("appconfig", configureSettings =>
{
configureSettings.Credential = credential;
}, configureOptions =>
{
configureOptions.ConfigureKeyVault(configureKv =>
{
configureKv.SetCredential(credential);
});
string instanceId = Environment.GetEnvironmentVariable("INSTANCE_ID") ??
throw new InvalidOperationException("The environment variable 'INSTANCE_ID' is not set or is empty.");
configureOptions.Select($"Instances:{instanceId}:RemoteGisUsername");
configureOptions.Select($"Instances:{instanceId}:RemoteGisPassword");
configureOptions.ConfigureRefresh(refreshOptions =>
{
refreshOptions.RegisterAll();
refreshOptions.SetRefreshInterval(TimeSpan.FromMinutes(5));
});
});
builder.Services.Configure<InstanceOptionsDictionary>(builder.Configuration.GetSection("Instances"));
Hey, @Fazer01
From the error messages you provided, it seems like the issue is not related to the Aspire integration.
Let me put some backgrounds here.
Aspire.Microsoft.Extensions.Configuration.AzureAppConfiguration is built on Microsoft.Extensions.Configuration.AzureAppConfiguration (which is our client library, let's call it configuration provider).
Microsoft.Extensions.Configuration.AzureAppConfiguration is built on Azure.Data.AppConfiguration` (which is our SDK). difference
From the error stack, aspire and configuration provider worked as expected. The configuration provider called the API from the SDK, the SDK sent the request and it got the response. It eventually failed at parse the response.
The error shows that you’re missing the NuGet dll your app needs at runtime:
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'System.Memory.Data, Version=8.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified. stderr F stderr F File name: 'System.Memory.Data, Version=8.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'
Could you check the runtime environment of Azure App Container app?
@Fazer01 I cannot reproduce this bug on my side. I successfully deployed my AppHost project to Azure Container Apps and the App Config component works well. I attached the app I used. I followed this doc and used Visual Studio to deploy it.
Google pointed me here while I was searching: "Could not load file or assembly 'System.Memory.Data, Version=8.0.0.1" The thing is that no such package exists, only 8.0.1.
On my side, I'm deploying ASP.NET Core 9 application to the Linux ARM64 server as a self-contained app.
The app crashes during the start-up phase, where FastEndpoins are trying to discover the endpoints using reflection.
When I publish the app, I have System.Memory.Data DLL in my publish folder, but it is version 6.0.x.
Everything is OK on my Windows machine. It started a couple of days ago.
Hey, @KUTlime Are you using Azure App Configuration SDK or its Aspire component?
Hey, @Fazer01
From the error messages you provided, it seems like the issue is not related to the Aspire integration.
Let me put some backgrounds here.
Aspire.Microsoft.Extensions.Configuration.AzureAppConfigurationis built onMicrosoft.Extensions.Configuration.AzureAppConfiguration(which is our client library, let's call it configuration provider).Microsoft.Extensions.Configuration.AzureAppConfiguration is built onAzure.Data.AppConfiguration` (which is our SDK). differenceFrom the error stack, aspire and configuration provider worked as expected. The configuration provider called the API from the SDK, the SDK sent the request and it got the response. It eventually failed at parse the response.
The error shows that you’re missing the NuGet dll your app needs at runtime:
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'System.Memory.Data, Version=8.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified. stderr F stderr F File name: 'System.Memory.Data, Version=8.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'
Could you check the runtime environment of Azure App Container app?
Hi there! Had holiday season so was not able to write a reply back in time.
I have checked the runtime environment which is displayed accessing the container (Azure Function container app) and running env:
ASPNET_VERSION=8.0.19
HOST_VERSION=4.1040.300.7
DOTNET_VERSION=8.0.19
FUNCTIONS_WORKER_RUNTIME_VERSION=9.0
Also we use .NET 9 as configured in the .csproj's TargetFramework property:
<TargetFramework>net9.0</TargetFramework>
Hope you have sufficient information. If not, please do not hesitate to reply.
Thanks in advance!
Hey, @Fazer01
I am not sure whether this is the root cause of the issue, but I just checked my container app, my container app is using .net 9.
May I ask how did you deploy your app to container app? I deployed it using visual studio. Could try the way in this doc? Also try to deploy the example I attached in the previous comment.
Hey, @zhiyuanliang-ms
As stated we are using .NET Aspire 9.4.0 and .NET 9. The deployment is done by using azd-tooling in our Azure Devops Pipeline.
I've just deleted the specific functionapp container from my container app environment and did a redeploy (ran the pipeline again so azd provision and azd deploy are executed. For the provisioning there is nothing much to be done as all infrastructure is already there. The azd deploy command creates and pushes new images to be used when creating and spinning up a container.
When the new revision is created (based on the new image), I bashed into the container app, but it still says the following: I will use a new resourcegroup where I deploy the application to to check if a new provisioning will solve this issue.
Regards,
Hey, @Fazer01 Sorry for the convenience!
As stated we are using .NET Aspire 9.4.0 and .NET 9. The deployment is done by using azd-tooling in our Azure Devops Pipeline.
I have checked the runtime environment which is displayed accessing the container (Azure Function container app) and running env: ASPNET_VERSION=8.0.19 HOST_VERSION=4.1040.300.7 DOTNET_VERSION=8.0.19 FUNCTIONS_WORKER_RUNTIME_VERSION=9.0
But you have checked the runtime env, it is using .net 8. This is weird. So I guess whether it is because azd-tool has a bug. I suppose that azd should configure the container app to use .net 9. Is it possible to use another way to deploy your application or make sure that the container app uses .net 9. Because I successfully deployed my app and the only difference we know here is the .net sdk version, so I want to see whether it can solve the problem, if you use the same deployment way(or even the same app).
I've also just started randomly run into this in the last few hours (13 hours ago there was a successful run). Extracting the file out of my container image gives me this:
The app is using the latest .NET 9 SDK and targets .NET 9. It doesn't use Aspire.Microsoft.Extensions.Configuration.AzureAppConfiguration, but it does use Aspire 9.4.1 and Microsoft.Azure.AppConfiguration.AspNetCore.
I'm don't know how/where this 6.0.36 version of System.Memory.Data is coming from yet.
It only seems to affect the app when it's running as a container - the normal CI is fine, and so is F5 debugging in Visual Studio with Aspire. I'm only getting issues in a workflow where I run the app as part of a Docker Compose file to validate OTel metrics.
https://github.com/martincostello/costellobot/pull/2675/commits/afead80ceef596f67d90963aa5e2793c2e739bed is the commit I'm seeing the problem with.
The failures that lead me to be investigating things locally with my container usage is an unrelated in another container image I'm using, but that doesn't explain the first error I got when investigating which was above. I made that error go away by manually upgrading System.Memory.Data to 9.0.8. I still don't know where the 6.0.36 version came from.
Hey, @martincostello
I guess this is because the Azure App Configuration client libraries (i.e.. Microsoft.Azure.AppConfiguration.AspNetCore and Microsoft.Extensions.Configuration.AzureAppConfiguration) are using the old versions of .net libraries.
I opened an issue to track it.
@zhiyuanliang-ms It's hard to say, it's a bit of a guessing game of if it will fix if we can't reproduce the issue. As far as I'm aware, there shouldn't be anything wrong with how we are referencing packages.
I digged into the dependency trees for a while and found some interesting things.
Microsoft.Extensions.Configuration.AzureAppConfiguration depends on Azure.Data.AppConfiguration 1.6.0 which depends on Azure.Core 1.44.1.
However, the latest version of Azure.Data.AppConfiguration is 1.6.1, which depends on Azure.Core [1.46.0]
However, 1.46.0 has been unlisted. This is interesting.
Then, I checked the Azure.Core 1.47.1, it depends on System.Memory.Data 8.0.1. PR
@zhiyuanliang-ms This is inline with what I said about it being a bit hard to pin down without a repro on our side. Originally it was thought that the cause would be the 'old versions of .net libraries', now investigation leads one to believe that the culprit is the Azure.Data.AppConfiguration package. However, it's hard to confirm.
I'm not against proactively updating outdated package versions where appropriate, but it is not clear it will fix the issue.
I'm having the same error, but when trying to connect to an Event Hub instance using EventHubProcessor publishing on Azure Container Apps.
Microsoft.Extensions.Hosting.Internal.Host[11] Hosting failed to start System.IO.FileNotFoundException: Could not load file or assembly 'System.Memory.Data, Version=8.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
File name: 'System.Memory.Data, Version=8.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'
at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateEventHubsConnectionAsync(CancellationToken cancellationToken)
at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateProcessingPreconditions(CancellationToken cancellationToken)
at Azure.Messaging.EventHubs.EventProcessorClient.ValidateProcessingPreconditions(CancellationToken cancellationToken)
at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingInternalAsync(Boolean async, CancellationToken cancellationToken)
at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingAsync(CancellationToken cancellationToken)
at Azure.Messaging.EventHubs.EventProcessorClient.StartProcessingInternalAsync(Boolean async, CancellationToken cancellationToken)
at Azure.Messaging.EventHubs.EventProcessorClient.StartProcessingAsync(CancellationToken cancellationToken)
at FleetManagementPlatform.Blackbox.Texa.EventHub.EventHubConsumerWorker.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\Robertoc\source\repos\evai\fleet-management-platform\src\FleetManagementPlatform.Blackbox.Texa\EventHub\EventHubConsumerWorker.cs:line 41
at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__14_1(IHostedService service, CancellationToken token)
at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
Unhandled exception. System.InvalidOperationException: The event processor is already running and needs to be stopped in order to perform this operation. at Azure.Messaging.EventHubs.EventProcessorClient.EnsureNotRunningAndInvoke(Action action) at Azure.Messaging.EventHubs.EventProcessorClient.remove_ProcessEventAsync(Func`2 value) at FleetManagementPlatform.Blackbox.Texa.EventHub.EventHubConsumerWorker.Dispose() in C:\Users\Robertoc\source\repos\evai\fleet-management-platform\src\FleetManagementPlatform.Blackbox.Texa\EventHub\EventHubConsumerWorker.cs:line 52 at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.DisposeAsync() --- End of stack trace from previous location --- at Microsoft.Extensions.Hosting.Internal.Host.DisposeAsync() at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host) at FleetManagementPlatform.Microservices.TexaService.Program.Main(String[] args) in C:\Users\Robertoc\source\repos\evai\fleet-management-platform\src\FleetManagementPlatform.Microservices.TexaService\Program.cs:line 111
This is straight from the Aspire console of a deployed worker which reads an event hub.
EDIT: The Aspire version of all packages is 9.4.2
Also, i add in a separate comment, i had this exact same issue when executing this line of code against an existing container in Azure Blobs Storage (always Aspire 9.4.2):
await containerClient.CreateIfNotExistsAsync(cancellationToken: cancellationToken);
I don't know if this is related at all, but since the EventProcessorClient use the storage to persist checkpoints, maybe it's a thing. is there any workaround?
EDIT: Tried to downgrade everything to 9.3.1, but still nothing changes... is it perhaps something on Microsoft side?
Hey, @Fazer01
Could you try the latest 9.5 Aspire.Microsoft.Extensions.Configuration.AzureAppConfiguration and see whether this issue will disappear?
@zhiyuanliang-ms We didn't get the dependency update in yet. It'll be in a patch of 9.5.
Thanks for all the effort @zhiyuanliang-ms and @davidfowl! Any update on when we can expect a (patch)release of 9.5?
This issue has been hitting my team recently as well, except that we dont use Aspire -- we do use msbuild's native container publishing support, which I expect Aspire is also using? We have a handful of projects where certain build machines can get into a state where the published container's /app/System.Memory.Data.dll is version 6.0.0.0, as confirmed by ildasm:
except dotnet nuget why confirms that our project never depends on 6.0.0, only 8.0.1, even transitively. We were able to work around the issue by running dotnet clean on the impacted projects and then dotnet nuget locals all --clear to blow away our local nuget package cache. Re-publishing after those steps normally(?) works, but it does introduce a long delay in our inner loop.
Hey, @jackhorton Sorry for the inconvenience.
Which version of Microsoft.Extensions.Configuration.AzureAppConfiguration are you using? Does the issue still apply for 8.4.0?
@zhiyuanliang-ms we don't depend on Microsoft.Extensions.Configuration.AzureAppConfiguration at all:
> dotnet nuget why .\Swarm.Scheduler\ Microsoft.Extensions.Configuration.AzureAppConfiguration
Project 'Swarm.Scheduler' does not have a dependency on 'Microsoft.Extensions.Configuration.AzureAppConfiguration'.
It seems like this is more of a .NET SDK bug rather than a bug with this specific package -- I know thats not super helpful for an issue in the dotnet/aspire repo, but I am not sure how to get proof of that or which repo is actually responsible for the underlying issue. I got here purely by searching for the "could not find assembly" shape of error. If you have any tips for understanding this issue more deeply, I would be happy to collect more data and file a bug somewhere else to make this more actionable.
I see this got moved to the Aspire 13.1 milestones... do we have any idea on if the fix will be retroactive in any way?
