Large number of SQL connections requested from Workflow Core

[vasicvuk]

Describe the bug

We have configured 3 instances of Workflow Core together with Redis for Queue and Locking. We use SQL Server Database with EF Core, and we experienced performance issues regarding slow processing of Workflows and deadlocks. We have set 25 max concurrent workflows.

In performance counters (dotnet-counters) we got this info

Name    Current Value
[Microsoft.Data.SqlClient.EventSource]                                                                                                                                                                                                             
Rate of connections retrieved from the connection pool (count/sec / 1 sec)     580    
Rate of connections returned to the connection pool (count/sec / 1 sec)    568  

There are many connection requests from Workflow Core, can we optimize this? We also tried different numbers of max concurrent workflows, but workflow processing is prolonged.

Also with the growth of the tables, it becomes very slow to handle from handling 12 workflows per second to query querying the same amount per minute.

Is there any way to improve performance?