konga icon indicating copy to clipboard operation
konga copied to clipboard

KONGA goes down after a few minutes when connected to postgres

Open davidcv5 opened this issue 5 years ago • 13 comments

I setup KONGA in a k8s cluster in development mode with local storage and was working fine, but users and config gets deleted if a pod restarts, so I connected it to a postgres database in azure, and now after a few minutes, it just throws an error and dies. It keeps showing logs just fine until it fails:

debug: KongProxyController:listAllEntityRecords:req.method GET
debug: KongProxyController:listAllEntityRecords:req.url /plugins?size=1000
debug: KongProxyController:listAllEntityRecords:entity plugins
debug: KongService: listAllCb http://my-admin:8001/plugins?size=1000
debug: KongProxyHooks:afterEntityList called() plugins
events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at TLSWrap.onread (net.js:622:25)

Please help!

davidcv5 avatar Apr 10 '19 21:04 davidcv5

@davidcv5 , what version of Kong & Konga are you using?

pantsel avatar Apr 10 '19 21:04 pantsel

@pantsel 0.14.3.. pantsel:latest docker image. Thank you! My environment is like this:

NODE_ENV:    development
DB_ADAPTER:  postgres
DB_URI:  postgresql://user:password@host:5432/konga?ssl=true

davidcv5 avatar Apr 10 '19 21:04 davidcv5

@davidcv5 , same thing happens if you run Konga on production mode?

pantsel avatar Apr 10 '19 21:04 pantsel

@pantsel yes, I only changed it to development to see if I could get any logs, in production mode the error doesn't show, just crashes. Added the configuration in the comment above.

davidcv5 avatar Apr 10 '19 21:04 davidcv5

It seems to be some kind of TLS related error with ssl=true. For some reason the connection to the db gets reset or killed by postgress.

It seems it has something to do with the underlying pg driver or/and connection settings rather than Konga itself.

Have you tried increasing max_connections to your pg instance and see what happens?

I will need to try and reproduce it in order to understand what's what.

pantsel avatar Apr 10 '19 21:04 pantsel

@pantsel not exactly sure I can increase that.. I'm using Azure Database for PostgreSQL from the following list I'm using a General Purpose - 2 vCores - 150 max_connections https://docs.microsoft.com/en-us/azure/postgresql/concepts-limits Any other recommendation? Thanks you!!

davidcv5 avatar Apr 10 '19 22:04 davidcv5

@pantsel having this same problem. External postgres seems to work after initial installation, but after an hour or less it loses connection completely and doesn't re-establish.

After this happens all of the UI components ( routes, services, etc ) just load forever never returning any results.

One note I'm not using the ssl=true parameter. I am using Amazon Aurora Postgres in RDS.

Jinkxed avatar May 17 '19 17:05 Jinkxed

Having the exact same issue:

events.js:183 throw er; // Unhandled 'error' event Error: read ECONNRESET at TLSWrap.onread (net.js:622:25)

DigitalTargit avatar Jun 16 '19 20:06 DigitalTargit

I am using Azure Postgres and have same issue. For me it goes down immediately.

Logs -

Hook:api_health_checks:process() called
Hook:health_checks:process() called
Hook:start-scheduled-snapshots:process() called
Hook:upstream_health_checks:process() called
Hook:user_events_hook:process() called
User had models, so no seed needed
Start scheduled health checks for node 1
Kongnode had models, so no seed needed
Emailtransport seeds updated
-------------------------------------------------------
:: Fri Aug 16 2019 04:43:50 GMT+0000 (UTC)
Environment : production
Port        : 1337
-------------------------------------------------------
events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at TLSWrap.onread (net.js:622:25)


This works when SSL is turned off.

swapnild2111 avatar Aug 16 '19 04:08 swapnild2111

Similar problem with konga 0.14.9

Did someone find a solution by now without turning TLS off?

syscall: 'read' code: 'ECONNRESET', errno: 'ECONNRESET', at processTicksAndRejections (internal/process/task_queues.js:84:21) { at emitErrorAndCloseNT (internal/streams/destroy.js:60:3) at emitErrorNT (internal/streams/destroy.js:92:8) at TLSSocket.EventEmitter.emit (domain.js:482:12) at TLSSocket.emit (events.js:310:20) at TLSSocket. (/app/node_modules/sails-postgresql/node_modules/pg/lib/connection.js:94:12) at Connection.EventEmitter.emit (domain.js:482:12) at Connection.emit (events.js:310:20) at Connection. (/app/node_modules/sails-postgresql/node_modules/pg/lib/client.js:174:19) at Client.EventEmitter.emit (domain.js:482:12) at Client.emit (events.js:310:20) at Client. (/app/node_modules/sails-postgresql/node_modules/pg/lib/pool.js:39:20) at Object.EventEmitter.emit (domain.js:482:12) at Object.emit (events.js:310:20) Emitted 'error' event on PG instance at: at TLSWrap.onStreamRead (internal/stream_base_commons.js:205:27) Error: read ECONNRESET ^ throw er; // Unhandled 'error' event events.js:287

csmarvz avatar Oct 28 '20 08:10 csmarvz

I believe https://github.com/pantsel/konga/issues/645 is related to this issue

mloskot avatar Dec 06 '20 21:12 mloskot

I got the same problem when running konga:0.14.9 on Kubernetes cluster with the DB_SSL: true. If I remove the DB_SSL, it works well.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: konga-deployment
  namespace: konga
spec:
  replicas: 1
  selector:
    matchLabels:
      app: konga
  template:
    metadata:
      labels:
        name: konga
        app: konga
    spec:
      containers:
      - name: konga
        image: pantsel/konga:0.14.9
.......

ctrongminh avatar Apr 06 '21 04:04 ctrongminh

Any clue of this? I'm facing this error with Konga pod

Using postgres DB Adapter. Failed to connect to DB Error: read ECONNRESET at TCP.onStreamRead (internal/stream_base_commons.js:205:27) { errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }

fertab avatar Aug 14 '22 21:08 fertab