nestjs icon indicating copy to clipboard operation
nestjs copied to clipboard

Published 20 hours ago • verified publisher icon golevelup

Multiple RabbitSubscribe in one class, one looses connection

Open wodka opened this issue 2 years ago • 8 comments

We have a class similar to this:

class Message {
  @RabbitSubscribe({
    exchange: 'message',
    queue: 'message-a-queue',
    queueOptions: {
      durable: true,
      maxPriority: 10,
    },
    routingKey: 'message-a',
  })
  async handleA() {}

  @RabbitSubscribe({
    exchange: 'message',
    queue: 'message-b-queue',
    queueOptions: {
      durable: true,
      maxPriority: 10,
    },
    routingKey: 'message-b',
  })
  async handleB() {}
}

after a few days running then message-a-queue consumer disconnects! There is nothing logged and I have no clue how to debug this. To me it looks like only the later one reconnects.

wodka avatar Nov 04 '22 16:11 wodka

Same problem!

zimiovid avatar Dec 01 '22 17:12 zimiovid

Same for me -_-

Andr-07 avatar Dec 01 '22 17:12 Andr-07

@wodka Could you provide a small repo (the same exact golevelup modules version) so that we can see what's going on?

underfisk avatar Dec 14 '22 16:12 underfisk

I can try - problem is that this is only happening after running for a few days :/

wodka avatar Dec 14 '22 16:12 wodka

I can try - problem is that this is only happening after running for a few days :/

I believe there must be something that the module router that we have read might be throwing in which case the connection might be killed and not re-connected but I'd like to see if at some point I can play around with your repro and kinda attempt to force one handler to go down. If at some point you can give us more context, that would be great but feel free to create the repro when you can

underfisk avatar Dec 14 '22 16:12 underfisk

we have a similar setup and also experience similar issues, in our case: this.amqpConnection.publish('EVENT_QUEUE', 'tx-created', {...}); throws an error

throw new Error('AMQP connection is not available');
/app/node_modules/@golevelup/nestjs-rabbitmq/lib/amqp/connection.js:192

this causes nestjs service to restart (if no catch handler exists) and seems like there has been a memory leak since then.

EvgeniLeonti avatar Jan 02 '23 09:01 EvgeniLeonti

:/ trying to get it to reproduce I've now found one problem that was earlier not logged:

Error: Channel closed by server: 404 (NOT-FOUND)

-> the Channel exits and is filling and with a restart of the service process the entries...

more detailed logs 
Failed to setup a RabbitMQ channel - name: AmqpConnection / error: Channel ended, no reply will be forthcoming Error: Channel ended, no reply will be forthcoming
    at rej (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:201:7)
    at C._rejectPending (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:207:42)
    at C.toClosed (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:171:8)
    at C._closeChannels (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:394:18)
    at C.toClosed (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:401:8)
    at C.onSocketError (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:355:10)
    at Connection.emit (node:events:513:28)
    at Socket.go (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:481:12)
    at Socket.emit (node:events:513:28)
    at emitReadable_ (node:internal/streams/readable:590:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)"}


Failed to setup a RabbitMQ channel - name: AmqpConnection / error: Operation failed: QueueDeclare; 404 (NOT-FOUND) with message "NOT_FOUND - home node 'rabbit@rabbitmq-0.rabbitmq.rabbitmq-35547411-production.svc.cluster.local' of durable queue 'pigeon-email-transactional-queue' in vhost '/' is down or inaccessible" Error: Operation failed: QueueDeclare; 404 (NOT-FOUND) with message "NOT_FOUND - home node 'rabbit@rabbitmq-0.rabbitmq.rabbitmq-35547411-production.svc.cluster.local' of durable queue 'pigeon-email-transactional-queue' in vhost '/' is down or inaccessible"
    at reply (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:134:29)
    at C.accept (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:417:7)
    at Connection.mainAccept [as accept] (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:64:33)
    at Socket.go (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:478:48)
    at Socket.emit (node:events:513:28)
    at emitReadable_ (node:internal/streams/readable:590:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)



Failed to setup a RabbitMQ channel - name: AmqpConnection / error: Channel ended, no reply will be forthcoming Error: Channel ended, no reply will be forthcoming
    at rej (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:201:7)
    at C._rejectPending (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:207:42)
    at C.toClosed (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:171:8)
    at C._closeChannels (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:394:18)
    at C.toClosed (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:401:8)
    at C.onSocketError (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:355:10)
    at Connection.emit (node:events:513:28)
    at Socket.go (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:481:12)
    at Socket.emit (node:events:513:28)
    at emitReadable_ (node:internal/streams/readable:590:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)



Successfully connected to RabbitMQ broker (default)
Error: Channel closed by server: 404 (NOT-FOUND) with message "NOT_FOUND - home node 'rabbit@rabbitmq-0.rabbitmq.rabbitmq-35547411-production.svc.cluster.local' of durable queue 'pigeon-email-transactional-queue' in vhost '/' is down or inaccessible"
    at C.accept (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:422:17)
    at Connection.mainAccept (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:64:33)
    at Socket.go (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:478:48)
    at Socket.emit (node:events:513:28)
    at emitReadable_ (node:internal/streams/readable:590:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)



Unhandled rejection Error: Channel ended, no reply will be forthcoming
    at rej (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:201:7)
    at C._rejectPending (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:207:42)
    at C.toClosed (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:171:8)
    at C._closeChannels (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:394:18)
    at C.toClosed (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:401:8)
    at C.onSocketError (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:355:10)
    at Connection.emit (node:events:513:28)
    at Socket.go (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:481:12)
    at Socket.emit (node:events:513:28)
    at emitReadable_ (node:internal/streams/readable:590:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)
Unhandled rejection Error: Channel ended, no reply will be forthcoming
    at rej (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:201:7)
    at C._rejectPending (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:207:42)
    at C.toClosed (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/channel.js:171:8)
    at C._closeChannels (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:394:18)
    at C.toClosed (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:401:8)
    at C.onSocketError (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:355:10)
    at Connection.emit (node:events:513:28)
    at Socket.go (/usr/src/app/node_modules/@golevelup/nestjs-rabbitmq/node_modules/amqplib/lib/connection.js:481:12)
    at Socket.emit (node:events:513:28)
    at emitReadable_ (node:internal/streams/readable:590:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)

Edit: not sure that error is really related, I keep trying

wodka avatar Jan 02 '23 20:01 wodka