moleculer

moleculer copied to clipboard

Hello.

Even in idle state, some messages are not removed from the queue that causes memory to go out of the bounds.

I see that on average 3 messages are sent to the queue per second. On the other hand, no application is consuming the messages.

It seems that amqp transporter makes a "memory leak" somehow.

@icebob Do you have any thoughts on that? Is there any configuration to prevent that?

RabbitMQ stores messages while they are not consumed. I think it is not a transporter issue, it is a configuration problem, but I don't know very well RabbitMQ.

Maybe @Nathan-Schwartz?

@katsanva It sounds like the issue isn't memory, but that you weren't expecting AMQP to store your messages. In the screenshot you shared it looks like only 1/3rd of the memory is being used.

AMQP will store your messages until you process or purge them. This is an intended behavior to make message delivery more reliable.

If this behavior isn't helpful for your use case, it may be best to switch to a different transporter like NATS :)

@Nathan-Schwartz @icebob Why should rabbitmq store heartbeat and ping-pong messages? IMO they should be removed after a small timeout.

Ahh, I see what you mean @katsanva . I thought you were talking about requests and such. I agree that ping-pongs and hearbeats (etc) should be discarded.

The current solution is to automatically delete those queues whenever the consumer disconnects. Since each node would have its own HEARTBEAT queue, I would expect them to be deleted as soon as a broker disconnects, so those messages wouldn't pile up.

What packet was associated with the queue in your screenshot?

Have you had issues with HEARTBEATS, PING, PONG, DISCOVER, DISCONNECT, or INFO packets piling up?

If there is a case not reasonably covered by auto-deletion, a default expiration time could be set by setting a default message timeToLive here: https://github.com/moleculerjs/moleculer/blob/master/src/transporters/amqp.js#L244

That would do the trick I think, though it is possible to run into a little weirdness because of this behavior:

Only when expired messages reach the head of a queue will they actually be discarded (or dead-lettered). Consumers will not have expired messages delivered to them.

• https://www.rabbitmq.com/ttl.html#per-message-ttl-caveats

An alternative approach would be to set a policy to enforce a queue limit for those queues (haven't tried this).

Even though there are a lot of HEARTBEAT/etc messages enqueued, my experience has been that brokers process them extremely quickly, and it hasn't been an issue for me in the past.

HEARTBEATS, PING, PONG, DISCOVER, DISCONNECT, or INFO

Yes, all these messages cause memory to grow.

Thanks in advance for your response @Nathan-Schwartz . I will probably try to move to NATS, instead of hacking amqp as far it is working not the way it should be.

But unfortunately, that is not a good solution if messaging is already hard-bind with amqp...

@katsanva thanks for the response. I haven't been able to produce this behavior before, so any steps and configuration necessary to reproduce the issue would be helpful. Most helpful would be a PR with a failing test that you would expect to pass.

If you want to stick with the AMQP transporter, you can put in a PR with the fix I mentioned above (I don't have the bandwidth atm).

In my experience switching between transporters has been very slick, although you would need a new way to interface with AMQP (or another queueing solution) if you are in need of one. This approach has the benefit of making it very clear where and how queues are being used, which can be abstracted away by the transporter at times.

Any updates about it?

If my company will stick to the rabbitmq and moleculer we will need to fix this problem and will create a PR for that. In other case I will probably make PR in my free time.

Any updates about this issue?

This issue was resolved by adding the autoDeleteQueues and expires options for service queues.