Hangfire dashboard throwing 'Could not resolve assembly' exception

[remus-corneliu]

I have setup a hangfire service with a server and a dashboard and I've deployed this to IIS , I also created a Db called "HangfireServerDb" and I've started the IIS server. The hangfire server started and created the require schema and tables and also for a period all jobs ran without any issues.

The IIS app pool is configured with

Start Mode : AlwaysRunning Recycling > Regular Time Interval : 0 Managed Pipeline Mode : Integrated

Hangfire version : 1.8.9

However at some point I can see in the dashboard the following exception

`System.InvalidOperationException: Recurring job can't be scheduled, see inner exception for details.
 ---> Hangfire.Common.JobLoadException: Could not load the job. See inner exception for the details.
 ---> System.IO.FileNotFoundException: Could not resolve assembly 'HangfireDashboard.WebApi'.
   at System.TypeNameParser.ResolveAssembly(String asmName, Func`2 assemblyResolver, Boolean throwOnError, StackCrawlMark& stackMark)
   at System.TypeNameParser.ConstructType(Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
   at System.TypeNameParser.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
   at System.Type.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError)
   at Hangfire.Common.TypeHelper.DefaultTypeResolver(String typeName)
   at Hangfire.Storage.InvocationData.DeserializeJob()
   --- End of inner exception stack trace ---
   at Hangfire.Storage.InvocationData.DeserializeJob()
   at Hangfire.RecurringJobEntity..ctor(String recurringJobId, IDictionary`2 recurringJob, ITimeZoneResolver timeZoneResolver, DateTime now)
   --- End of inner exception stack trace ---
   at Hangfire.Server.RecurringJobScheduler.ScheduleRecurringJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, RecurringJobEntity recurringJob, DateTime now)`

And I am not sure of the reason. There are no changes whatsoever between in the deployment , it is the exact same code, and and there is no reason it should complain about the 'HangfireDashboard.WebApi' since the website points the this location

public class Startup
    {
        public IConfiguration _configuration;

        public Startup(IConfiguration configuration)
        {
            _configuration = configuration;
        }

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddControllersWithViews();

            services.ConfigureHealthchecks(_configuration);
            services.TryAddSingleton<ITimeZoneResolver>(new TimeZoneResolver());

            services.AddHangfire(configuration =>
            {
                configuration
                    .SetDataCompatibilityLevel(CompatibilityLevel.Version_180)
                    .UseSimpleAssemblyNameTypeSerializer()
                    .UseRecommendedSerializerSettings()
                    .UseSqlServerStorage(
                        _configuration["ConnectionStrings:HangfireConnectionString"],
                        new SqlServerStorageOptions
                        {
                            CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
                            SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
                            QueuePollInterval = TimeSpan.Zero,
                            UseRecommendedIsolationLevel = true,
                            DisableGlobalLocks = true
                        })
                        .UseFilter(new AutomaticRetryAttribute
                        {
                            Attempts = int.TryParse(_configuration["HangfireSettings:AutomaticRetry:Attempts"] ?? "5", out int attempts) ? attempts : 5,
                            OnAttemptsExceeded = AttemptsExceededAction.Fail,
                            DelaysInSeconds = Enumerable.Repeat((int)TimeSpan.FromSeconds(int.TryParse(_configuration["HangfireSettings:AutomaticRetry:DelaysInSeconds"] ?? "60", out int delaysInSeconds) ? delaysInSeconds : 60).TotalSeconds, attempts).ToArray()
                        });
            });

            services.AddHangfireServer(options =>
            {
                options.Queues = new[]
                {
                    "accounts",
                    "dictionaries",
                    "liki-24",
                    "product-label-manager",
                    "tv-campaigns",
                    "web-platform",
                };
                options.ServerTimeout = TimeSpan.FromMinutes(5);
            });

            if (!string.IsNullOrWhiteSpace(_configuration["Liki24:Api:BaseUri"]))
            {
                services.AddHttpClient<Liki24HttpClient>(
                    httpClient =>
                    {
                        httpClient.BaseAddress = new Uri(_configuration["Liki24:Api:BaseUri"]);
                        httpClient.DefaultRequestHeaders.Clear();
                        httpClient.DefaultRequestHeaders.Add(HeaderNames.Accept, "application/json");
                    });
            }

            if (!string.IsNullOrWhiteSpace(_configuration["WebPlatform:Api:BaseUri"]))
            {
                services.AddHttpClient<WebPlatformHttpClient>(
                    httpClient =>
                    {
                        httpClient.BaseAddress = new Uri(_configuration["WebPlatform:Api:BaseUri"]);
                        httpClient.DefaultRequestHeaders.Clear();
                        httpClient.DefaultRequestHeaders.Add(HeaderNames.Accept, "application/json");
                    });
            }

            if (!string.IsNullOrWhiteSpace(_configuration["ProductLabelManager:Api:BaseUri"]))
            {
                services.AddHttpClient<ProductLabelManagerHttpClient>(
                    httpClient =>
                    {
                        httpClient.BaseAddress = new Uri(_configuration["ProductLabelManager:Api:BaseUri"]);
                        httpClient.DefaultRequestHeaders.Clear();
                        httpClient.DefaultRequestHeaders.Add(HeaderNames.Accept, "application/json");
                    });
            }

            if (!string.IsNullOrWhiteSpace(_configuration["TvCampaigns:Api:BaseUri"]))
            {
                services.AddHttpClient<TVCampaignsHttpClient>(
                    httpClient =>
                    {
                        httpClient.BaseAddress = new Uri(_configuration["TvCampaigns:Api:BaseUri"]);
                        httpClient.DefaultRequestHeaders.Clear();
                        httpClient.DefaultRequestHeaders.Add(HeaderNames.Accept, "application/json");
                    });
            }

            if (!string.IsNullOrWhiteSpace(_configuration["Dictionaries:Api:BaseUri"]))
            {
                services.AddHttpClient<DictionariesHttpClient>(
                    httpClient =>
                    {
                        httpClient.BaseAddress = new Uri(_configuration["Dictionaries:Api:BaseUri"]);
                        httpClient.DefaultRequestHeaders.Clear();
                        httpClient.DefaultRequestHeaders.Add(HeaderNames.Accept, "application/json");
                    });
            }

            if (!string.IsNullOrWhiteSpace(_configuration["Accounts:Api:BaseUri"]))
            {
                services.AddHttpClient<AccountsHttpClient>(
                    httpClient =>
                    {
                        httpClient.BaseAddress = new Uri(_configuration["Accounts:Api:BaseUri"]);
                        httpClient.DefaultRequestHeaders.Clear();
                        httpClient.DefaultRequestHeaders.Add(HeaderNames.Accept, "application/json");
                    });
            }
        }

        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            app.UseHangfireDashboard("/hangfire", new DashboardOptions
            {
                Authorization = new[] { new HangfirePassThroughAuthorizationFilter() },
            });

            app.UseMiddleware<ErrorHandlerMiddleware>();
            app.UseMiddleware<SerilogMiddleware>();
            app.UseHttpsRedirection();
            app.UseRouting();
            app.UseAuthentication();
            app.UseAuthorization();
            app.UseCors("CorsPolicy");

            app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllers();

                endpoints.MapHealthChecks("/hc", new HealthCheckOptions()
                {
                    Predicate = x => true,
                    ResponseWriter = HealthcheckExtensions.WriteHealthCheckResponse,
                    AllowCachingResponses = false,
                    ResultStatusCodes =
                    {
                        [HealthStatus.Healthy] = StatusCodes.Status200OK,
                        [HealthStatus.Degraded] = StatusCodes.Status200OK,
                        [HealthStatus.Unhealthy] = StatusCodes.Status503ServiceUnavailable
                    }
                });

                endpoints.MapHealthChecks("/liveness", new HealthCheckOptions
                {
                    Predicate = r => r.Name.Contains("self")
                });
            });

            AccountsJobs.AddJobs(_configuration,
                new RecurringJobOptions
                {
                    TimeZone = TimeZoneInfo.Local
                });

            DictionariesJobs.AddJobs(_configuration,
                new RecurringJobOptions
                {
                    TimeZone = TimeZoneInfo.Local
                });

            Liki24Jobs.AddJobs(_configuration,
                new RecurringJobOptions
                {
                    TimeZone = TimeZoneInfo.Local
                });

            ProductLabelManagerJobs.AddJobs(_configuration,
                new RecurringJobOptions
                {
                    TimeZone = TimeZoneInfo.Local
                });

            TvCampaignsJobs.AddJobs(_configuration,
                new RecurringJobOptions
                {
                    TimeZone = TimeZoneInfo.Local
                });

            WebPlatformJobs.AddJobs(_configuration,
                new RecurringJobOptions
                {
                    TimeZone = TimeZoneInfo.Local
                });
        }

Is this a known issues or something ? I've been strugling with this for almost one week and I cannot understand why it does behave as such.

Further investigations

Application: w3wp.exe
CoreCLR Version: 6.0.2623.60508
.NET Version: 6.0.26
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AggregateException: One or more hosted services failed to stop. (A task was canceled.)
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Hangfire.Processing.TaskExtensions.WaitOneAsync(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token) in C:\projects\hangfire-525\src\Hangfire.Core\Processing\TaskExtensions.cs:line 96
   at Hangfire.Processing.BackgroundDispatcher.WaitAsync(TimeSpan timeout, CancellationToken cancellationToken) in C:\projects\hangfire-525\src\Hangfire.Core\Processing\BackgroundDispatcher.cs:line 82
   at Hangfire.Server.BackgroundProcessingServer.WaitForShutdownAsync(CancellationToken cancellationToken) in C:\projects\hangfire-525\src\Hangfire.Core\Server\BackgroundProcessingServer.cs:line 148
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at HangfireDashboard.WebApi.Program.Main(String[] args) in D:\CI_Data\TeamCity\buildAgent2\work\ea4fb37391903d0f\Services\HangfireDashboard\Program.cs:line 22
   at HangfireDashboard.WebApi.Program.<Main>(String[] args)

[vipasane]

Any changes on your running environment? I have faced some when uninstalling obsolete hosting bundles since some extensions are targeting older .net versions. Used this tool to track down those troublemakers: https://github.com/ironfede/NetVersionFinder