DiskQueue icon indicating copy to clipboard operation
DiskQueue copied to clipboard

Threads vs Tasks

Open pracsol opened this issue 1 year ago • 1 comments

Hi, this looks very useful for a project I'm involved in, compared to rolling my own very basic persistent file queue. Your examples are running these read and write operations in separate threads. If these sessions were opened and flushed in async tasks, do you foresee any additional risks around locking/contention, which wouldn't be present if I was using separate threads? I'm basically imagining two tasks like the following, based on your examples in the Readme.md:

protected override async Task ExecuteAsync(CancellationToken stoppingToken) 
{
    var tasks = new[]
    {
        Task.Run(() => RunHourlyPolling(stoppingToken),
        Task.Run(() => RunUploader(stoppingToken),
        // several other tasks doing other things.. 
    };
    Task.WaitAll(tasks, stoppingToken);
}
protected async Task RunHourlyPolling(CancellationToken stoppingToken) 
{
    while (someBoolFlag && !stoppingToken.IsCancellationRequested) 
    {
        using (var session = queue.OpenSession())
        {
	        session.Enqueue(GetSomethingFromSomewhere());
	        session.Flush();
        }
        await Task.Delay(anHourInMs, stoppingToken);
    }
}
protected async Task RunUploader(CancellationToken stoppingToken)
{
    while (!stoppingToken.IsCancellationRequested) 
    {
        using (var session = queue.OpenSession()) {
            var data = session.Dequeue();
	    if (data == null) {
                // widen out the delay here so the synchronous call for the dequeue op isn't blocking other tasks constantly
                Task.Delay(1000, stoppingToken); 
                continue;
            }
	    try 
            {
	        MaybeDoWork(data)
		session.Flush();
            }
            catch (RetryException) 
            {
	        continue;
	    }
            catch 
            {
	        session.Flush();
	    }
	}
    }
}

I guess I can see that the enqueue and dequeue methods are implemented to run synchronously, and that implementation seems appropriate, but unless I'm widening out that Task.Delay sufficiently, the queue monitoring loop is going to be getting in the way of everybody else quite often, right? So it seems advisable to run this as a separate thread, which would then let me implement the dequeue call on a 100ms delay like you have in the example with little risk of impacting other operations. If I can afford to wait a second or multiple seconds between queue checks, to give plenty of room to other tasks, are there other considerations I should be thinking of with this kind of implementation?

pracsol avatar May 16 '24 21:05 pracsol

Hi. The examples are running in different threads mainly to prove that it works that way. DiskQueue is fine to run inside async methods, or to run synchronously in a single thread.

i-e-b avatar May 17 '24 07:05 i-e-b