nestjs

nestjs copied to clipboard

After some time running my application for some reason I keep getting a connection loss.

This is the error it shows:

[Nest] 369295 - 06/02/2024, 10:13:51 PM ERROR [AmqpConnection] Disconnected from RabbitMQ broker (default) Error: Unexpected close at succeed (/home/guidiamond/Work/juri-api/node_modules/amqplib/lib/connection.js:229:15) at onOpenOk (/home/guidiamond/Work/juri-api/node_modules/amqplib/lib/connection.js:211:7) at /home/guidiamond/Work/juri-api/node_modules/amqplib/lib/connection.js:114:11 at /home/guidiamond/Work/juri-api/node_modules/amqplib/lib/connection.js:107:11 at Socket.recv (/home/guidiamond/Work/juri-api/node_modules/amqplib/lib/connection.js:456:9) at Object.onceWrapper (node:events:633:28) at Socket.emit (node:events:519:28) at Socket.emit (node:domain:488:12) at emitReadable_ (node:internal/streams/readable:832:12) at processTicksAndRejections (node:internal/process/task_queues:81:21)

Right after that I get: /home/guidiamond/Work/juri-api/node_modules/amqplib/lib/channel.js:369 throw new IllegalOperationError(msg, stack); ^ IllegalOperationError: Channel closed at ConfirmChannel.<anonymous> (/home/guidiamond/Work/juri-api/node_modules/amqplib/lib/channel.js:369:11) at ConfirmChannel.nack (/home/guidiamond/Work/juri-api/node_modules/amqplib/lib/channel_model.js:221:10) at requeueErrorHandler (/home/guidiamond/Work/juri-api/node_modules/@golevelup/nestjs-rabbitmq/src/amqp/errorBehaviors.ts:27:11) at /home/guidiamond/Work/juri-api/node_modules/@golevelup/nestjs-rabbitmq/src/amqp/connection.ts:507:19 at processTicksAndRejections (node:internal/process/task_queues:95:5)

The thing is I'm running this locally and the only connection that gets lost is with rabbitmq, and for some reason it doesn't try to reconnect after that. Does anybody know what could be the issue here?

This is what I added to imports in my API's module:

   RabbitMQModule.forRoot(RabbitMQModule, {
      exchanges: [
        {
          name: Exchanges.API,
          type: 'direct',
          options: {
            durable: true,
          },
        },
        {
          name: Exchanges.DEAD_LETTER,
          type: 'direct',
          options: {
            durable: true,
          },
        },
      ],
      channels: {
        [Channels.MAPEIA_V1]: {
          prefetchCount: 15,
          default: true,
        },
        [Channels.RESTORE_V2]: {
          prefetchCount: 2,
          default: false,
        },
      },
      queues: [
        {
          name: Queues.PROCESSA_V1_DLQ,
          routingKey: Queues.PROCESSA_V1_DLQ,
          exchange: Exchanges.DEAD_LETTER,
          createQueueIfNotExists: true,
        },
        {
          name: Queues.RESTORE_V2_DLQ,
          routingKey: Queues.RESTORE_V2_DLQ,
          exchange: Exchanges.DEAD_LETTER,
          createQueueIfNotExists: true,
        },
      ],
      uri: `amqp://${config.getTyped('rabbitMq.username')}:${config.getTyped(
        'rabbitMq.password',
      )}@${config.getTyped('rabbitMq.hostname')}:${config.getTyped(
        'rabbitMq.port',
      )}`,
      enableControllerDiscovery: true,
    }),

Same here!!

I ran into this issue on a development machine. RabbitMQ log showed at the same time:

2024-07-22 01:26:05.192698+00:00 [error] <0.8046.1> closing AMQP connection <0.8046.1> ([::1]:52036 -> [::1]:5672):
2024-07-22 01:26:05.192698+00:00 [error] <0.8046.1> missed heartbeats from client, timeout: 5s
2024-07-22 01:26:16.201197+00:00 [warning] <0.8043.1> closing AMQP connection <0.8043.1> ([::1]:52026 -> [::1]:5672, vhost: '/', user: 'rabbitmq'):
2024-07-22 01:26:16.201197+00:00 [warning] <0.8043.1> client unexpectedly closed TCP connection
2024-07-22 01:26:16.201654+00:00 [warning] <0.8040.1> closing AMQP connection <0.8040.1> ([::1]:52024 -> [::1]:5672, vhost: '/', user: 'rabbitmq'):
2024-07-22 01:26:16.201654+00:00 [warning] <0.8040.1> client unexpectedly closed TCP connection
2024-07-22 01:26:16.201921+00:00 [warning] <0.8037.1> closing AMQP connection <0.8037.1> ([::1]:52022 -> [::1]:5672, vhost: '/', user: 'rabbitmq'):
2024-07-22 01:26:16.201921+00:00 [warning] <0.8037.1> client unexpectedly closed TCP connection

It seems that if the node process is stuck and no heartbeat is sent, the AMQP connection gets closed.

Solution for me was to increase heartbeat setting and put timeouts to external HTTP requests.

uri: 'amqp://...@...:5672?heartbeat=20',
connectionManagerOptions: { heartbeatIntervalInSeconds: 20 },

This issue is stale because it has been open for 30 days with no activity.

The issue seems to be related with amqlib, the underlying lib. As @zerobits-de suggested, increasing heartbeat interval may help with the issue