Hangfire
Hangfire copied to clipboard
HangfireIO
Support for `async` background job methods
Heavy I/O bound background jobs consume worker thread inefficiently – instead of doing the real work they wait for the completion of I/O operation(s). Currently if an application has a lot of such background jobs, it is better to increase the worker pool size to process jobs more efficiently. However increased worker pool causes performance problems for CPU-bound jobs.
Instead of waiting for the I/O completion, it would be useful to yield the worker thread to process another background job instead, for example, using the default mechanism with async and await keywords:
public async Task IOBoundMethod()
{
var documents = await _documentsService.FetchAll();
documents.Process();
}
But we should ensure that the completion is being called inside the Hangfire's worker thread, and not in CLR's thread pool thread (to not to compete with request processing pipeline, use additional storage connections and so on). Another consideration should be dedicated to job filters – CaptureCultureAttribute changes the thread context, and it should be changed again on each continuation. There are also problems with reliability (outstanding jobs should be requeued), shutdown events (outstanding operations should be canceled), and perhaps with something else.
+1 this would be very nice (http://discuss.hangfire.io/t/async-task-jobs/73)
As of now Hangfire does not protect you from incorrectly using async methods as background tasks. I.e. you can schedule an async operation which will run successfully (however, Hangfire will not await the result so esceptions will be lost + job execution statistics will be incorrect). If async support is far down the road, how about throwing an exception when trying to schedule/run a job which returns a Task?
@Isantipov, this is a brilliant proposal, will schedule it for next minor release
Wow, such a popular feature, want it too :-) This feature requires a lot of breaking changes and it is postponed to the version 2.0, that will be released in the next year with async methods for storages as well.
@odinserj, do you already have any idea about the async method you need from the storage ?
Hi @odinserj - any idea when async will be here. Will definitely be moving up to the pro version when this happens?
+1
...except I'm not sure how you're going to pull off keeping it limited to pro. In my mind only two things need to be done:
- Add overloads that accept
Expression<Func<Task>>, including those variations that accept a generic instance argument, ~~and variations that acceptCancellationTokeninstances as well~~ it looks like there is already support forCancellationToken - Add the bit of code that waits for the
Task, if any, to complete and catches anyAggregateException~~(possibly with some flattening.)~~ (Link to relevant code)
Alright, so I've submitted a PR that adds support. I should note that for the time being, it's really not a big deal to go without this support -- all one needs to do is provide both SomeMethod and SomeMethodAsync:
public void SomeMethod()
{
try
{
SomeMethodAsync().Wait();
}
catch (AggregateException aggregate)
{
throw aggregate.InnerException;
}
}
public async Task SomeMethodAsync()
{
await ...
}
A little bit of churn, sure, but then "actual" async support requires one to put in those #pragma warning disable 4014 statements or deal with compiler warnings. Oh well. :beers:
