[BUG] Connecting to the DB takes a very long time

[stheiven]

Describe the bug We have noticed that at times connecting to the DB take a very long time (e.g., 100s of milliseconds). I have attached an image of a Honeycomb trace where the telemetry data shows that the connect span takes 403 milliseconds.

Expected behavior I would expect that the connect takes 10s of milliseconds not 100s,

Environment (please complete the following information, because it helps us investigate better):

• OS: linux
• Version 1.1.1
• DB: Postgresql

[tolgaozen]

Hi @stheiven , could you provide a bit more information about the environment? Is the Permify instance in the same region as the database?

[stheiven]

@tolgaOzen Yes, they are in the Permify instance is in the same region as the DB. The connect times were reduced once we put a RDS proxy in front of the DB, but we still some spikes where the connect times are 100s of ms. We have increased the max connections on the Permify pods as well to see if this helps (still testing this), but I suspect that the connection spikes will still remain after this change.

[tolgaozen]

@stheiven , could you please share your config file after hiding confidential fields?

[stheiven]

@tolgaOzen I've added attached manifest for the pod. We had the best results with the default setting of 20 for PERMIFY_DATABASE_MAX_OPEN_CONNECTIONS

pod.txt

[omer-topal]

Hi @stheiven,

Since RDS Proxy improved things, the remaining spikes are the app opening new client connections when its pool has no idle ones to reuse. The “connect” span here is the app establishing a fresh TCP/TLS session to the proxy, which is slower than reusing an existing connection.

Please test with these parameters to keep a small warm pool on the app side:

- name: PERMIFY_DATABASE_MAX_IDLE_CONNECTIONS
  value: '10'
- name: PERMIFY_DATABASE_MAX_CONNECTION_IDLE_TIME
  value: '5m'