Continuations stuck in Awaiting State

[paulcaru]

trafficstars

Hi,

Any idea why continuation jobs are stuck in awaiting state? This is happening everything

Parent Job was always succeeded.

Setup 2 Clients and 1 Processing server (Hangfire 1.6.17) MySql Database

[odinserj]

This could be caused by MySql extension, if it doesn't have a proper implementation for transactions and distributed locks.

[alexjebens]

I am seeing the same Issue on MSSQL Express 2016 (13.0.4001.0) I have tried downgrading to 1.6.16,15,14 and 13 without success.

[paulcaru]

Hi. The issue was not due to mysql extension.

I finally found out that the problem was somehow related to the unity injection. I was doing it like this.

.RegisterType<JobStorage>(new InjectionFactory(c => JobStorage.Current))
           .RegisterType<IJobFilterProvider, JobFilterAttributeFilterProvider>(new InjectionConstructor(true))
            .RegisterType<IBackgroundJobFactory, BackgroundJobFactory>()
            .RegisterType<IRecurringJobManager, RecurringJobManager>()
            .RegisterType<IBackgroundJobClient, BackgroundJobClient>()
            .RegisterType<IBackgroundJobStateChanger, BackgroundJobStateChanger>()

Playing around with unity I managed to get it working but I dont know if maybe something else would be broken so I did a wrapper to the BackgroundJob class of hangfire so i leave the initializations up to hangfire when its called and then I just inject my wrapper class.

[rklfss]

Maybe a bit late, but we ran into the same issue. This always happened, if the next state as well is an AwaitingState whose parent has already finished (we build some kind of job trees where some AwaitingState chains exist when waiting for multiple jobs to finish).

We used this as a workaround and never got stuck in AwaitingState anymore.

public sealed class ExtendedContinuationsSupportFilterAttribute : JobFilterAttribute, IElectStateFilter, IApplyStateFilter
{
    private readonly ContinuationsSupportAttribute _continuationsSupportAttribute;

    public ExtendedContinuationsSupportFilterAttribute()
        : this(false)
    {
    }

    public ExtendedContinuationsSupportFilterAttribute(bool pushResults)
        : this(pushResults, new HashSet<string> { DeletedState.StateName, SucceededState.StateName })
    {
    }

    public ExtendedContinuationsSupportFilterAttribute(HashSet<string> knownFinalStates)
        : this(false, knownFinalStates)
    {
    }

    public ExtendedContinuationsSupportFilterAttribute(bool pushResults, HashSet<string> knownFinalStates)
    {
        _continuationsSupportAttribute = new ContinuationsSupportAttribute(pushResults, knownFinalStates);
        Order = _continuationsSupportAttribute.Order;
    }

    public void OnStateElection(ElectStateContext context)
    {
        // This is the fix! Should that be fixed in ContinuationsSupportAttribute?
        // Fix continuations when having an AwaitingState chain
        IState lastCandidateState;
        do
        {
            lastCandidateState = context.CandidateState;
            _continuationsSupportAttribute.OnStateElection(context);
        }
        while (lastCandidateState != context.CandidateState && context.CandidateState is AwaitingState);
    }

    public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
        _continuationsSupportAttribute.OnStateApplied(context, transaction);
    }

    void IApplyStateFilter.OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
        // nothing
    }
}

in configuration

internal static IGlobalConfiguration UseExtendedContinuationsSupportFilter(this IGlobalConfiguration options)
{
    return options.Use(
        new ExtendedContinuationsSupportFilterAttribute(),
        filter =>
        {
            var filters = GlobalJobFilters.Filters;

            // remove buggy ContinuationsSupportAttribute and replace with fix
            var continuationFilter = filters.FirstOrDefault(f => f.Instance is ContinuationsSupportAttribute);
            if (continuationFilter != null)
            {
                filters.Remove(continuationFilter.Instance);
            }

            filters.Add(filter);
        });
}