Deadlock in low thread environment

[MichalSznajder]

I use linq2db.EntityFramewortkCore (6.7.1) in my read layer in ASP MVC application. I limit amount of threads in ThreadPool with ThreadPool.SetMaxThreads(12, 100). Recently I witnessed that my application stops responding to any HTTP request. By taking dump I found that most threads blocks in linq2db (deadlock):

[HelperMethodFrame_1OBJ: 00007fbed4974db0] System.Threading.Monitor.ObjWait(Int32, System.Object)
System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ManualResetEventSlim.cs @ 570]
System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2985]
System.Threading.Tasks.Task.InternalWaitCore(Int32, System.Threading.CancellationToken) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2920]
System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].GetResultCore(Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs @ 467]
*** LinqToDB.EntityFrameworkCore.Internal.LinqToDBForEFQueryProvider`1[[System.__Canon, System.Private.CoreLib]].GetAsyncEnumerator(System.Threading.CancellationToken)
System.Runtime.CompilerServices.ConfiguredCancelableAsyncEnumerable`1[[System.__Canon, System.Private.CoreLib]].GetAsyncEnumerator() [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/ConfiguredCancelableAsyncEnumerable.cs @ 42]
Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions+<ToListAsync>d__65`1[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions+<ToListAsync>d__65`1[[System.__Canon, System.Private.CoreLib]], Microsoft.EntityFrameworkCore]](<ToListAsync>d__65`1<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[[System.__Canon, System.Private.CoreLib]](System.Linq.IQueryable`1<System.__Canon>, System.Threading.CancellationToken)
LinqToDB.EntityFrameworkCore.LinqToDBExtensionsAdapter.ToListAsync[[System.__Canon, System.Private.CoreLib]](System.Linq.IQueryable`1<System.__Canon>, System.Threading.CancellationToken)
LinqToDB.AsyncExtensions+<ToListAsync>d__8`1[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[LinqToDB.AsyncExtensions+<ToListAsync>d__8`1[[System.__Canon, System.Private.CoreLib]], linq2db]](<ToListAsync>d__8`1<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
LinqToDB.AsyncExtensions.ToListAsync[[System.__Canon, System.Private.CoreLib]](System.Linq.IQueryable`1<System.__Canon>, System.Threading.CancellationToken)
App.Database.Repositories.MoviesRepository+<GetMovies>d__2.MoveNext()
-- earlier frames omitted

This callstack originates from simple code similar to this:

var localResult = await sortQuery
    .AsNoTracking()
    .ToLinqToDB()
    .ToListAsyncLinqToDB();

Code investigation brings clear sync over async pattern that causes this in LinqToDBForEFQueryProvider

public IAsyncEnumerator<T> GetAsyncEnumerator(CancellationToken cancellationToken)
{
    return Task.Run(() => QueryProvider.ExecuteAsyncEnumerable<T>(Expression, cancellationToken),
    cancellationToken).Result.GetAsyncEnumerator(cancellationToken);
}

There was attempt to fix this issue in https://github.com/linq2db/linq2db.EntityFrameworkCore/commit/ea55615a655ae0b91e90bb8db7da15fcc4b4d546 but new fix does not help too much in my case.

I patched LinqToDBExtensionsAdapter.ToListAsync with this code:

public async Task<List<TSource>> ToListAsync<TSource>(
	IQueryable<TSource> source,
	CancellationToken token)
{
	var list = new List<TSource>();
	var provider = source.Provider as LinqToDBForEFQueryProvider<TSource>;
	var enumerator = await provider!.ExecuteAsyncEnumerable<TSource>(provider.Expression, token).ConfigureAwait(false);
	
	await foreach (var element in enumerator.WithCancellation(token))
	{
		list.Add(element);
	}

	return list;
}

and it immediately helps but I think this is a dirty hack.

Any idea how this can be fixed in another way?

[MaceWindu]

That's a good point actually. Never liked those Task.Run things in our code. We should review them and probably update to use SafeAwaiter helper

[MichalSznajder]

What to do you mean by SafeAwaiter? I don't see reference to this in 'linq2db' nor in 'linq2db.EFC`.

[MaceWindu]

helper we use in linq2db in some places https://github.com/linq2db/linq2db/blob/master/Source/LinqToDB/Async/SafeAwaiter.cs

[MichalSznajder]

It's not going to help here. LinqToDBForEFQueryProvider::GetAsyncEnumerator is a sync method. Any call to Task.Result or Task.Wait() will block and may lead to deadlock (or rather thread starvation like in my scenario).

ExpressionQuery.ExecuteAsyncEnumerable is async because of eager loading. It needs to load "preambles" before returning main query results. So there is clear sync-over-async issue.

Do you think when EFCore bridge is used this eager loading will happen?

[TheConstructor]

I guess #241 would fix this problem. I am still not sure, that it is a good/the best way to fix this. Can someone test, if I am right?