Timed out waiting for connection

[holzeis]

The coordinator is regularly failing to acquire a connection from the diesel pool manager.

{"timestamp":"2024-06-14T15:18:30.520400301Z","level":"ERROR","fields":{"message":"Failed to store dlc message. timed out waiting for connection","peer":"02aa6fc81e341d69b36174365644a3560bfcc51ce0fb9bb67d93712bc54117cbb8"},"target":"coordinator::dlc_handler"}

          This problem seems to be happening frequently

https://itchysats.grafana.net/explore?schemaVersion=1&panes=%7B%2291e%22:%7B%22datasource%22:%22grafanacloud-logs%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22grafanacloud-logs%22%7D,%22editorMode%22:%22builder%22,%22expr%22:%22%7Bcontainer%3D%5C%22coordinator%5C%22%7D%20%7C%3D%20%60timed%20out%20waiting%20for%20connection%60%20%7C%20json%22,%22queryType%22:%22range%22%7D%5D,%22range%22:%7B%22from%22:%22now-30d%22,%22to%22:%22now%22%7D%7D%7D&orgId=1

Originally posted by @holzeis in https://github.com/get10101/10101/issues/2628#issuecomment-2171826875