PostgreSQL running out of remaining connections

[lallinger-tech]

Environment info

• NooBaa Version: PR #6428

Actual behavior

After generating about 20k events some endpoints and the core are starting to show this error:

�[32mMar-31 11:04:18.611�[35m [Endpoint/12] �[31m[ERROR]�[39m core.util.postgres_client:: find failed { last_update: { �[32m'$gte'�[39m: �[33m1617188553553�[39m } } { projection: { last_update: �[33m0�[39m } } SELECT data #- '{last_update}' as data FROM tiers WHERE data->'last_update'>='1617188553553'::jsonb error: remaining connection slots are reserved for non-replication superuser connections
    at Parser.parseErrorMessage (/root/node_modules/�[4mnoobaa-core�[24m/node_modules/�[4mpg-protocol�[24m/dist/parser.js:278:15)
    at Parser.handlePacket (/root/node_modules/�[4mnoobaa-core�[24m/node_modules/�[4mpg-protocol�[24m/dist/parser.js:126:29)
    at Parser.parse (/root/node_modules/�[4mnoobaa-core�[24m/node_modules/�[4mpg-protocol�[24m/dist/parser.js:39:38)
    at Socket.<anonymous> (/root/node_modules/�[4mnoobaa-core�[24m/node_modules/�[4mpg-protocol�[24m/dist/index.js:10:42)
�[90m    at Socket.emit (events.js:315:20)�[39m
�[90m    at Socket.EventEmitter.emit (domain.js:467:12)�[39m
�[90m    at addChunk (internal/streams/readable.js:309:12)�[39m
�[90m    at readableAddChunk (internal/streams/readable.js:284:9)�[39m
�[90m    at Socket.Readable.push (internal/streams/readable.js:223:10)�[39m
�[90m    at TCP.onStreamRead (internal/stream_base_commons.js:188:23)�[39m {
  length: �[33m140�[39m,
  severity: �[32m'FATAL'�[39m,
  code: �[32m'53300'�[39m,
  detail: �[90mundefined�[39m,
  hint: �[90mundefined�[39m,
  position: �[90mundefined�[39m,
  internalPosition: �[90mundefined�[39m,
  internalQuery: �[90mundefined�[39m,
  where: �[90mundefined�[39m,
  schema: �[90mundefined�[39m,
  table: �[90mundefined�[39m,
  column: �[90mundefined�[39m,
  dataType: �[90mundefined�[39m,
  constraint: �[90mundefined�[39m,
  file: �[32m'postinit.c'�[39m,
  line: �[32m'828'�[39m,
  routine: �[32m'InitPostgres'�[39m
}

After restarting all pods, all endpoints and the core show the error. As far as i can tell everything still keeps working.

Expected behavior

1. No error

Steps to reproduce

1. Create a function
2. Create a bucket
3. Set a trigger on the bucket to fire the function on ObjectCreated event
4. Upload 10000-20000 objects (size does not matter) to the bucket

[lallinger-tech]

Could it be, that multiple endpoints worsen this? a fresh install with 10 endpoints already throws this error.. @liranmauda @jeniawhite

[dannyzaken]

@lallinger-arbeit this error is thrown by PostgreSQL when exceeding the max_connections which is the maximum number of concurrent connections the DB is configured to handle (see here). We are not changing the default which is 100 connections

for each process (endpoints included) we are using an internal connection pool with a max size of 10 connections, so multiple endpoints can definitely worsen it. When you first encountered it, how many endpoints were running? since each process should use at most 10 connections I wouldn't expect to see this before reaching a total of 10 processes (endpoints + 3 processes in noobaa-core)

[lallinger-tech]

@lallinger-arbeit this error is thrown by PostgreSQL when exceeding the max_connections which is the maximum number of concurrent connections the DB is configured to handle (see here). We are not changing the default which is 100 connections

so 10 endpoints would quite definitely exceed the current settings?

When you first encountered it, how many endpoints were running?

first i only used 3

[dannyzaken]

10 endpoints are likely to hit this issue, but with 3 endpoints we shouldn't get to more than 60 connections. It would be helpful if you can reproduce with 3 endpoints and attach the logs here (also DB logs). I will also try to reproduce it.

[lallinger-tech]

i'll try to reproduce it, but probably won't have time to do it until end of next week..

[lallinger-tech]

@liranmauda are you tracking the remaining connections issue here or somewhere else? if you track it somewhere else this can be closed

[liranmauda]

@dannyzaken ^^