trino-gateway icon indicating copy to clipboard operation
trino-gateway copied to clipboard

Add connection pooling to Trino Gateway backend datastore

Open rdsarvar opened this issue 8 months ago • 1 comments

Out-of-the-box JDBI does not seem to leverage connection pools and will create a new connection for every query against the backend datastore as seen in the public Slack channel here. During high traffic this can lead to reaching maximum connection counts against the backend datastore.

It has been noted that there are multiple ways of getting around the limitation of maximum number of connections:

  1. Optimize the underlying datastore configurations so that connections drop off quicker
  2. Use pgbouncer as a service in between Trino Gateway and PostgreSQL (assuming postgres is used as the backend store instead of MySQL or similar)
  3. Implement connection pooling as part of the Trino Gateway code itself. Example PostgresSQL connection pooling provided by JDBI docs: https://jdbi.org/#_high_availability

As an enhancement we are looking to minimize overall connections required by Trino Gateway against its backend datastore.

rdsarvar avatar Jun 03 '24 14:06 rdsarvar