Failed to change state to a 'Succeeded' one due to an OutOfMemoryException

[jesuissur]

Since upgrading to release 1.7.28 (from 1.7.7), Hangfire cannot update state when our app is under load. The task is completed with success but the task state is Failed. The symptom is also existing in the 1.7.27 release for us.

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
      at Hangfire.Common.SerializationHelper.Serialize(Object value, Type type, SerializationOption option)
         at Hangfire.Common.SerializationHelper.Serialize[T](T value, SerializationOption option)
            at Hangfire.Common.SerializationHelper.Serialize[T](T value)
               at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.SetJobState(String jobId, IState state)
                  at Hangfire.States.CoreStateMachine.ApplyState(ApplyStateContext context)
                     at Hangfire.States.StateMachine.ApplyState(ApplyStateContext initialContext)
                        at Hangfire.States.BackgroundJobStateChanger.ChangeState(StateChangeContext context)
                           at Hangfire.Server.Worker.TryChangeState(BackgroundProcessContext context, IStorageConnection connection, IFetchedJob fetchedJob, IState state, String[] expectedStates, CancellationToken initializeToken, CancellationToken abortToken)

This is the setup we use about serialisation

hangfireConfiguration.SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
                                              .UseSimpleAssemblyNameTypeSerializer()
                                              .UseRecommendedSerializerSettings();

Thanks, Phil

[jesuissur]

With the latest upgrade (1.7.29), Hangfire retries indefinitely the job that previously had the Failed to change state to a 'Succeeded' message on the 1.7.28 version.

Every 10-15 minutes a new state is added in the database for the job and each one is in "Processing". Our logs for that job also confirm the infinite retries by Hangfire.

[odinserj]

Hangfire can not do anything if a background job method throws this exception. If it ignores it, then application can potentially be in the wrong state. The only correct solution for this is to increase the available memory to the application or think how to avoid consuming too much memory.