Same long running job makes multiple instances of itself why?

[sseyalioglu]

Hi,

I have a scheduled job which kicks in every 30 minutes and it can take hours to complete. Due to that, before every execution, I check if same job is already running and skipping execution using monitor API (shown below)

    private static bool IsAlreadyQueuedOrProcessing(string jobName, string jobId)
    {
        var jobStatesToCheckAgainst = new[] {"Processing", "Enqueued", "Created"};
        var monitoringApi           = JobStorage.Current.GetMonitoringApi();
        var queue                   = monitoringApi.Queues().FirstOrDefault()?.Name ?? "default";
        var enqueuedJobs            = monitoringApi.EnqueuedJobs(queue, 0, 1000);
        var processingJobsAlready   = enqueuedJobs.Count(j => j.Key != jobId && j.Value.Job.Method.Name == jobName && jobStatesToCheckAgainst.Contains(j.Value.State));
        return processingJobsAlready > 0;
    }

But, somehow at the schedule times, hangfire by itself re executes the same running job (not the new one) and causes trouble to my process which I want to understand why.

Here is the proof:

[odinserj]

Switch to the recommended configuration – https://docs.hangfire.io/en/latest/configuration/using-sql-server.html#configuration – if you are on 1.7.X version, or just use the SqlServerStorageOptions.SlidingInsibilityTimeout to 5 minutes to use better fetching logic – I see you have 30 minutes between retries, and this may be related to connectivity problems.

However if you are using another storage, not SQL Server, than please consult in the related repository.

[sseyalioglu]

Hi, I use MySql and try recommendations in general. I am not sure which recommendation is the one helps to the situation.

Here is my initialization:

            services.AddHangfire(c =>
            {
                c.UseStorage(new MySqlStorage(hangfireConnection, new MySqlStorageOptions
                {
                    QueuePollInterval        = TimeSpan.FromSeconds(60),
                    PrepareSchemaIfNecessary = false,
                    DashboardJobListLimit    = 1000
                }));
                c.SetDataCompatibilityLevel(CompatibilityLevel.Version_170);
                c.UseRecommendedSerializerSettings();
            });

Is there something specific that causes this issue?

[Gillardo]

Hi,

I have a scheduled job which kicks in every 30 minutes and it can take hours to complete. Due to that, before every execution, I check if same job is already running and skipping execution using monitor API (shown below)

    private static bool IsAlreadyQueuedOrProcessing(string jobName, string jobId)
    {
        var jobStatesToCheckAgainst = new[] {"Processing", "Enqueued", "Created"};
        var monitoringApi           = JobStorage.Current.GetMonitoringApi();
        var queue                   = monitoringApi.Queues().FirstOrDefault()?.Name ?? "default";
        var enqueuedJobs            = monitoringApi.EnqueuedJobs(queue, 0, 1000);
        var processingJobsAlready   = enqueuedJobs.Count(j => j.Key != jobId && j.Value.Job.Method.Name == jobName && jobStatesToCheckAgainst.Contains(j.Value.State));
        return processingJobsAlready > 0;
    }

But, somehow at the schedule times, hangfire by itself re executes the same running job (not the new one) and causes trouble to my process which I want to understand why.

Here is the proof:

How are you scheduling you job and passing in the jobId?

[billpeace]

Please try using the SkipWhenPreviousJobIsRunningAttribute job filter from the referenced gist https:// gist.github.com/odinserj/a6ad7ba6686076c9b9b2e03fcf6bf74e