Performance Issue with active Console

[BenjaminAbt]

We have 170.000 jobs and this takes normally around 2 minutes. If we use Hangfire.Console it explodes to 17 minutes.

Every job writes around ~10 lines avg to the console. What is the best way to increase the performance to an acceptable value?

We're using MSSQL and right now we take some tests with a higher polling value.

[pieceofsummer]

Hello, Benjamin.

It seems that creating a lot of concurrent transactions causes a huge performance hit to your database.

I'm currently working on an optional feature to aggregate console write operations from multiple concurrent jobs and save them asynchronously in batches. This should reduce the number of transactions and improve overall performance.

[pieceofsummer]

Today I've actually done some tests with a simple job which does nothing other than writes to Console:

public static void TestJob2(PerformContext context)
{
    var progress = context.WriteProgressBar("test");
    
    for (var i = 0; i <= 100; i++)
    {
        context.WriteLine("Test {0}", i);
        progress.SetValue(i);
        Thread.Sleep(200);
    }
}

The following screenshot shows the result of its two consecutive runs on the same system. Note that I've been using a remote Sql Server instance for this test. And not a very good one, it seems — Hangfire running locally took ~150ms to show any of the dashboard pages. Because of that, both results include ~1s latency, so the actual numbers should be some 20.672 vs 53.551 seconds.

So, job # 64 is using the latest version of Hangfire.Console package, and it has actually wasted more than a half of time on database writes. The jobs performing lots of frequent and insignificant (and meaningless 😂) updates turn out to be a huge stress for Hangfire.

Job # 65, on the other hand, is a very first actual test of what it may become with a next major update. It features background database writes, combining updates from multiple jobs into a single transaction, and even dropping progress bar updates which are yet to be saved, but are already outdated.

Though there's still a long way to go before it becomes a thing.

[BenjaminAbt]

Thanks for your improvement. Any public branch I could review the code?

[pieceofsummer]

There are still some issues to address, like job console not updating after the job has changed state, but some lines are not written yet, or late lines are not assigned correct expiration time. I’ll let you know once it will be available for test.

[ben-tcw]

We're also running into this issue causing increased job time and db load. Do you have your fix available in a separate branch at all that we could assist with development on or perform any testing with?

[pieceofsummer]

@BenjaminAbt @ben-tcw I've just pushed beta sources into async-commit branch, so you may try it out.

By default it commits synchronously (as before). To enable asynchronous commits you'll need to provide additional background process when starting Hangfire server:

app.UseHangfireServer(additionalProcesses: new[] { new ConsoleBackgroundProcess() });

I'm very interested to hear feedback from you, as I've never tested it in real-world scenarios yet.

[ben-tcw]

That's brilliant Alexy. I'm taking a look now.

Is there anything required to disable the synchronous submission or does that happen automatically on adding the additional process?

[pieceofsummer]

@ben-tcw it will detect and use the background process automatically, falling back to synchronous commits if the background process is not registered.

[ben-tcw]

A first look at this branch suggests it does exactly what we need it to and offloads the writes so they're not clogging up connections etc... An interesting quirk being the HF dashboard only refreshes the console while the job is "running". But the async console message collector may submit console entries after the job has technically completed. This means you end up with a completed job without all the console lines on-screen. A page refresh corrects this.

[ascott18]

@pieceofsummer are there any plans to keep going with the work you started in the async-commit branch? Right now we really aren't able to use Hangfire.Console in some scenarios because of synchronous writes of the logs.

I appreciate all the work that you've put into this library nonetheless!

[oskary]

We've also hit this head first yesterday. Jobs started from nowhere running really slow and it was really hard to figure out what was making the slowdown. But at last I'm quite sure that it's this console-addon, which is wonderful. But it could really need some optimization.

We're turning off the console for now.